Data Import/Export Tasks Tips and Best Practices

If you are importing or exporting data frequently, automating the process via Data import/export tasks can save you time. Using Data import/export tasks, you can import files directly from online repositories such as FTP, SCP, and HTTP sites as well as Amazon S3. You can also export files to the same repositories or allow Data import/export tasks to send them to you via email.

However, there are several factors that may affect the operation and performance of Data import/export tasks. This article describes the best practices for ensuring that this feature runs in its optimum capability.

Note: Data import/export tasks are available in select plans. For more information, see Caspio Plans and Pricing.

Best Practices

• Data import/export tasks during low activity times, both by account users and app users. Ensure that your repository is fast and is capable of high-speed data transfer.

Note: During the import process, the performance of the destination table may degrade and the table may have to be locked partially or completely. These events can affect app users that are working with the same table.

• One common problem you might encounter after scheduling an import task is when the design of the future import files or the design of the destination table is modified.
• Do not schedule a task too frequently. Allow enough time for the task to completely finish before letting another recurrence of the same task to begin.
• Do not schedule any manual import or export of the same table or view.
• Your table may be growing and, as it incurs more data, a task that used to take only a few minutes to run may gradually require longer processing time. We recommend that you perform a task manually to see exactly how long it takes to complete. Taking the growth of your table into consideration, schedule it in such a way that it provides adequate time between tasks. Also, note that the processing time of the task depends on the speed of your repository.
• Occasionally review the logs to address any issues that may have occurred during the processing of tasks. Mouse-over a task on the Data import/export tasks screen and click History. On the next screen, mouse-over the log you wish to view and click Properties. These logs are also sent to you by email in real time.

Other Considerations

Mission Critical Import/Exports

Many factors can affect the ability to run your tasks on time. The best practices provided in this article can help you in scheduling your tasks, but if your import/export is highly time-sensitive you have two other options:

• Use the Caspio API for real-time changes to your data.
• Sign up for a Caspio Enterprise Plan, which provides a dedicated infrastructure for your organization.

Scheduled Time

The data import/export tasks feature do its best to run your task at the time you have set. However, there are no guarantees that this is possible due to the factors discussed below. Since many customers schedule tasks exactly at the top of the hour (i.e., 8:00, 11:00, 1:00), we recommend that you schedule your tasks at odd hours (i.e., 9:15, 7:45, 12:30). This increases the chance that your task starts on time and completes quickly.

Maximum Simultaneous Tasks

Currently, Data import/export tasks allow up to three tasks per account to run simultaneously. Additional tasks are queued to run after one of your task has finished. On Caspio Enterprise accounts, this number is flexible.

Expired/Skipped Tasks

Tasks that cannot begin within 15 minutes from their scheduled time are considered expired and are skipped. This can happen when your account has too many tasks and some tasks cannot begin until more than 15 minutes from their original scheduled time has passed.

For example, there are six tasks scheduled to run at 11:00 AM. The system starts tasks 1, 2, and 3 at the scheduled time. Assuming task 1 and 2 finish at 11:05, the system starts tasks 4 and 5. Assuming task 3 finished at 11:16, the system skips task 6 because at 11:16, task 6 is already expired.

Data Transfer

Any data that goes into or out of your account incurs data transfer—this includes the Data import/export tasks feature. Make sure that your Plan contains adequate data transfer allocation for your Data import/export tasks activity.

Resource Usage

Data transfer utilization

Data import/export tasks utilize data transfer. All imports and exports of data utilize data transfer. Data import/export tasks automate the process and allow you to schedule it and run it on a recurring basis. If scheduled to operate too frequently and if your files are large you will see a noticeable increase in your data transfer usage.

See our pricing page or contact your account manager for more information on choosing the right plan.

Backing up the objects prior to import

Backup accrues data transfer equal to the exported file size when the file is placed in the Messages area for download. No additional data transfer occurs no matter how many times you download the file.

Exporting and importing

Import and export using Data import/export tasks incur data transfer. However, import and export from the Tools menu in the account, do not incur data transfer.

Email utilization

Data import/export tasks have the ability to generate notification emails when a task completes or fails. Unlike our current plans that offer unlimited emails, older plans are limited. Note that these emails count towards your total monthly limit.

See our pricing page or contact your account manager for more information on choosing the right plan.

Common Error and Warning Messages

Pay close attention to the details of the Data import/export tasks error and warning messages to understand the issue and see how they can be resolved. Below is a list of the most common error and warning messages.