Automatic data import & automatic deletion
Automatic data import keeps your monitoring list up to date without anyone lifting a finger: your business software drops the CSV file onto our FTPS server, and we read it in automatically every night. The same route can also remove records from monitoring automatically – using an additional deletion file.
Introduction
We of course support automated data import – it is the only way to reduce the effort of sanctions list and PEP screening to a minimum while still fully meeting your legal obligations. Two approaches have proven themselves in practice: using our API, which lends itself particularly well to online shops (see also the API documentation), and the automatic import of CSV files (more on how the CSV file should be structured / OWASP compliance), which is described below.
The same route also lets you remove records from monitoring automatically – for that we additionally configure a separate deletion file (see Deleting records automatically).
Scenario
You use an ERP, CRM or other business system and want to automate the upload of the CSV files, so that nobody has to keep your records in easycompliance up to date by hand.
How it works
First, make sure your software supports the automated export of CSV files. This is a basic feature of every common solution, or can at least be added afterwards via what is usually called workflows or batch runs. Save the generated CSV file wherever you like. What matters is that the file always keeps the same name and that the name contains no timestamp or similar. You can of course generate several CSV files, for example "private-customers.csv", "business-customers.csv" and "suppliers.csv". If you would like to use automatic import, please contact our customer service – they will provide your credentials and set the automatic import up for you.
Providing the file via FTPS
The generated CSV file is then uploaded to an online storage location every evening. You get access to it via FTPS, because we collect the CSV files from there automatically every night and start the import. A direct automatic import into easycompliance is not possible, and neither is us collecting the files from one of your own servers.
The easiest route is probably an automated upload using the free tool WinSCP. Every administrator knows this program, and automating file uploads with it takes only a few minutes (described by WinSCP themselves, or see below for our ready-made example configuration). You are of course free to use a different tool. Some ERP systems – particularly cloud-based ones – can even transfer the files they generate themselves.
IMPORTANT: Please upload your files by 10 p.m. (CET/CEST) so that they can still be picked up by the next import run. Otherwise they will be delayed by one day.
Reserved namespace: the EXPORT_*.csv namespace in the root folder
is reserved for the optional automatic CSV export of your screening results (see the
section below) – please choose different names for your own files, especially import
source files.
Connection settings for your FTP client
In your client, choose the connection type "FTP over TLS/SSL" or "explicit FTPS". Access via SFTP is not possible! SFTP is an entirely different protocol that our server does not offer – an SFTP connection attempt will therefore always fail.
| Setting | Value |
|---|---|
| Server | ftps.easycompliance.de |
| Protocol | FTP over TLS/SSL – explicit FTPS, not SFTP |
| Port | 21 |
| User name / password | Provided by customer service |
| Transfer mode | Passive mode may be required (see below) |
| Target folder | Exactly the directory agreed with us – usually the login or home directory. Some setups use a configured subfolder; never change the location without checking with us first |
Passive mode: Depending on your client, you may have to switch on passive mode. Many programs use it by default; if the connection is established but the directory listing stays empty or the transfer breaks off, active mode is almost always the cause. Switch your client to "passive" and open the ports listed below.
A note on the SSL certificate: Depending on your client, a warning about the SSL certificate may appear, stating that the certificate is "not trusted". In that case you have to configure your client to trust the certificate. Please note that the certificate is reissued every three months – your client has to survive that change, otherwise the upload will silently stop working after the next renewal.
There are two ways to get there:
| Option | When it makes sense |
|---|---|
| Pin the fingerprint (recommended) – enter the SHA-256 fingerprint of our server certificate in your client (WinSCP shows it on the first manual connection). | When you want full protection. The fingerprint changes with every renewal, so you update it every three months; customer service will tell you the current one. |
| Trust any certificate – this is how our ready-made WinSCP package is preconfigured, so the upload survives the certificate changes unattended. | When the upload has to run without supervision. The transfer stays TLS-encrypted, but the check that our server is really the one answering is skipped – this setting offers no protection against an active man-in-the-middle. |
If your client accepts the certificate without any warning, there is nothing to do at all – regular certificate validation applies, and that is preferable to both options above.
A note on ports and firewalls
In passive mode the client and the server negotiate the ports themselves, so normally no firewall changes are needed on your side. If the connection does not work, open the port range 40000:40099 and make sure your firewall does not block outbound connections.
For security reasons we use several firewalls and proxies, so it can happen that you are unable to establish an FTPS connection to us at first. In that case please contact our customer service and let us know the IP address you transfer the data from. We will then unblock that IP manually.
Deleting records automatically
When a business partner leaves your books, they usually no longer need to be screened. That can be automated as well – using exactly the same mechanism as the import: you additionally upload a deletion file to the same FTPS account, and we process it automatically every night to remove the records it contains from your monitoring. The finer points – what happens per run, and how several files interact – are under Good to know.
How it works:
- Alongside the import file, your software exports a second CSV file containing the records to be deleted, for example "delete.csv".
- You upload that file together with the import file to the same FTPS account (same folder, same time window – by 10 p.m.).
- Customer service configures this file once as a deletion file. Together we decide whether records are removed by name (default) or by reference.
- From then on the deletion runs automatically every night – with nothing further to do on your side.
Good to know:
- The import file and the deletion file are always two separate files. A single file cannot import and delete at the same time; each file has a fixed name agreed with us.
- The deletion file is always processed BEFORE your import files each night. If a record appears both in the deletion file and (again) in your import file – for example a re-onboarded business partner while the deletion file has not been updated yet – the import therefore wins: the record is removed first and re-imported right afterwards, so it stays in monitoring.
- The same rules apply to the deletion file as to the import file: always the same file name (no timestamp), an unchanged column structure and the same delimiter. If any of this changes, please tell customer service in advance.
- Deleting by reference: there is no fallback to the name. Once deletion by reference is set up, records are matched on the reference only. Rows without a reference are skipped, and rows with a mismatching reference delete nothing – the record would silently stay in monitoring. Make sure every row of your deletion file carries a reference that matches the one stored in easycompliance exactly. If you cannot guarantee that, deleting by name (the default) is the more robust choice.
- Deletion cannot be undone. The record disappears from active monitoring and you cannot bring it back – a record removed by mistake has to be imported and screened again.
- Each row removes one record. One row of the deletion file deletes exactly one matching record per run. If your monitoring list holds several entries with the same name (or the same reference), the rest disappear one at a time over the following nights. So a name you deliberately re-added can stay visible, because that night's deletion hits one of the duplicates instead. If you added it by hand in the customer portal and your imports are split into sub-lists, it will even stay for good (see the next point).
- If your imports are split into sub-lists, each file only affects its own sub-list. With several files, customer service can set up that separation. Where it is active, a deletion file removes records only from the sub-list it belongs to – an entry with the same name coming from another file is left untouched and has to be removed through that file's deletion file. Records you added or imported yourself in the customer portal belong to no sub-list and are never picked up by a deletion file – remove those in the portal. Without the separation, import and deletion act on your entire monitoring list. Customer service can tell you whether your imports are split.
- Keep the two files free of overlap anyway. If the same record appears in both the import file and the deletion file, the fixed order (delete first, then import) keeps it in monitoring – but deletion and import then work against each other needlessly every night. Only export records to the deletion file once you have removed them from the import file.
- The deletion file is processed on every run – even if you do not upload it again. We do not delete the file from the FTPS account after fetching it; as long as it sits there, the records it contains are removed again every night. Already-deleted entries are simply skipped. If you add one of them back to your import file, it stays – deletion runs before the import, the import wins. A record you added by hand in the customer portal, by contrast, is not in your import file: without the sub-list separation it is removed again every night, with the separation it stays for good (see above). So overwrite the file with your current departures; if deletions should stop for good, ask customer service to deactivate the deletion file.
- Only delete what really has to go. Check your software's export before the initial setup – ideally with just a few test records.
If you would rather delete once, by hand, use the file import for deletion in the customer portal – see Deleting records via file import.
Automatic CSV export of your screening results (optional)
The route also works in the other direction: on request, we provide the CSV status report of your monitoring list automatically every day on the same FTPS account – one file EXPORT_<segment>.csv per list segment, records without a segment in EXPORT_monitoring.csv. The files cover all modules, include a machine-readable hit column (JSON) and are overwritten on every run under fixed names; outdated export files (for example after a segment was renamed) are cleaned up automatically. Your software can therefore fetch and process the results from stable file names.
Activation is handled by customer service; an existing FTPS access is required. The EXPORT_*.csv namespace in the root folder is reserved for this – your own files (especially import source files) must not use these names.
Automating the upload with WinSCP
We have prepared ready-made scripts for you. All you have to do is enter the FTPS credentials you received from our customer service and specify the folder on your server that holds the files to be uploaded. The connection type, the port and the automatic handling of the certificate are already configured correctly.
Is a subfolder agreed for you? Then credentials and a local folder are not enough: without adjustment the script always uploads to the login/home directory, while the nightly fetch looks for the file in the agreed subfolder – it would not find it there and would skip the import silently. Add a cd <subfolder> line before the put line in the configuration file, or ask customer service for a version tailored to you. In the standard case (files go straight into the home directory) there is nothing to do.
Important – we have to activate the import: once the upload runs on your side, please reply to the email containing your credentials, or write to our customer service. We then check your files and switch the import on. Only after that are the uploaded files processed – until then they simply sit on the server.
Downloading the scripts and the instructions
You can download the scripts as a ZIP archive here: Go to download
The ZIP archive contains the following files:
| File | Description |
|---|---|
ANLEITUNG.md |
Read this first. Instructions for correct use and setup (German). |
easycompliance-upload.bat |
WinSCP execution script used by the scheduler. |
easycompliance-upload.txt |
Configuration file in which you have to change the user name, password and path. |
zeitplan-einrichten.bat |
Creates a scheduled Windows task that runs the upload at a set time. |
zeitplan-entfernen.bat |
Removes that scheduled Windows task again. |
Frequently asked questions
Which connection type do I have to configure in my FTP client?
Choose "FTP over TLS/SSL" or "explicit FTPS" on port 21, server ftps.easycompliance.de. Customer service provides your user name and password.
Can I upload the files via SFTP? No. Access via SFTP is not possible. Only FTPS (FTP over TLS/SSL, explicit) is supported. SFTP is a different protocol that our server does not offer.
Do I have to switch on passive mode? Possibly – it depends on your client. Many programs use passive mode by default. If the connection is established but the file list stays empty or the transfer breaks off, switch your client to "passive" and open ports 40000:40099.
My client reports that the certificate is "not trusted". What should I do? This warning can appear depending on your client. Configure your client to trust the certificate – most securely by pinning the SHA-256 fingerprint of our server certificate. Because the certificate is reissued every three months, update the fingerprint accordingly; customer service will tell you the current one. If the upload has to run unattended, you can instead let your client trust any certificate (this is how the WinSCP package is preconfigured) – the transfer stays TLS-encrypted but no longer protects against an active man-in-the-middle.
The upload worked for months and suddenly fails. Why? The most common cause is the SSL certificate, which is reissued every three months: if your client still pins the fingerprint of the old certificate, it will refuse the connection after the renewal. Update it to the current fingerprint (customer service will tell you) or configure the client to trust any certificate – the latter is more convenient but gives up protection against an active man-in-the-middle. If the problem persists, check your credentials and firewall and contact customer service.
How do I remove records from monitoring automatically? Via a separate deletion file: your software additionally exports a CSV file containing the records to be deleted, you upload it to the same FTPS account, and customer service configures it once as a deletion file. The records it contains are then removed automatically every night – one matching record per row and run, always before your import files are processed. Records you created yourself in the customer portal are not picked up while your imports are split into sub-lists; delete those in the portal.
Can I combine import and deletion in a single file? No. Import and deletion always run through two separate files, each with a fixed file name. If the same record appears in both files, the fixed order (delete first, then import) means the import wins – but keep the files free of overlap anyway.
Can an automatic deletion be undone? No. The record disappears from active monitoring and you cannot bring it back. A record deleted by mistake has to be imported and screened again.
Are records deleted by name or by reference? By name by default. On request we set up deletion by reference – for that, the reference has to exist and match in both the deletion file and your records in easycompliance. In that mode there is no fallback to the name: rows without a reference, or with a mismatching one, are skipped and the record stays in monitoring.
Can the automatic import create duplicate records? Only if duplicate detection is not configured for your import – in that case the contents of the file are created again on every nightly run, even if you did not upload the file again. With duplicate detection, records that already exist are skipped; it is on by default for newly configured imports. Note, though: if your imports are split into sub-lists, detection only looks within the sub-list that this file feeds – so the same name can appear once in each sub-list and be screened more than once. Customer service can tell you how your import is configured. For background on detection by name or by reference, see Avoiding duplicate entries on import.
By what time do I have to upload my files? By 10 p.m. (CET/CEST). Files uploaded later are only picked up by the following night's import, which delays them by one day.
What happens if a file is not uploaded one day? Then we process the last uploaded version again the following night. We do not delete your files from the FTPS account after fetching them, and the import does not check whether the file is new – it simply reads whatever is there. What this means for the import file depends on whether duplicate detection is configured for your automatic import (it is on by default for newly configured imports):
- With duplicate detection: records that already exist are skipped (within the relevant sub-list, where imports are split), so the repeated run is largely harmless. If you removed a name from monitoring by hand in the meantime and it is still in the file, it will be added back.
- Without duplicate detection: the contents of the file are created again on every run. Your monitoring list then gains another complete copy every night, inflating both its size and the number of results.
The deletion file runs again as well and removes the records it contains once more – including any you deliberately re-added. The exception is records you created yourself in the customer portal while your imports are split into sub-lists: a deletion file never picks those up. So upload your files with current content every day if you can. If you do not know how your import is configured, customer service can check it for you.
May the file name or the structure of the file change later on? No. The file name, the column order and the delimiter have to stay unchanged – timestamps in the file name are not possible either. New records (rows) may be added at any time. If you do need to change the name or the structure, please tell customer service in advance.
Do I need API access for the automatic import? No. The automatic CSV import via FTPS works without an API connection. The API is an alternative for real-time integrations, for example in online shops.
Related articles
- Structuring the CSV files for import
- Automatic import (set up by customer service)
- Deleting records via file import
Last reviewed: