Behind the Build: An Independent Developer's Journey with ftpClient
In the DolphinDB ecosystem, independent developers are emerging as a vital force — transforming niche operational challenges into production-grade solutions that drive real efficiency gains. These aren't theoretical tools. They come directly from production environments, built by practitioners who've experienced the pain points firsthand.
One such contributor is a seasoned data systems developer who built ftpClient to solve a persistent workflow problem: reliable, efficient file transfer across heterogeneous systems. His plugin delivers:
- Fast, reliable data sync for backup distribution and cross-platform file collection
- Elimination of redundant I/O by removing the need for intermediate storage
- Native DolphinDB integration that simplifies cross-system data pipelines and increases automation
A Conversation with Developer Flossie
Q: Could you introduce yourself?
A: I'm Flossie, an independent developer with extensive experience in data systems. My work focuses on hybrid C++/Python development, cross-system network protocols, and DolphinDB performance optimization—particularly around large file transfers and incremental data synchronization in financial environments.
Q: What inspired you to build ftpClient?
A: It came from a real pain point in quantitative trading. Our team regularly synchronized massive volumes of market data from multiple remote FTP servers into DolphinDB for analysis. The process was fragmented: download via FileZilla or WinSCP, convert formats, then manually import into DolphinDB.
For batch transfers, this workflow was error-prone and incredibly inefficient. I wanted to solve this for our team—and add native FTP capabilities to the DolphinDB ecosystem. That's how ftpClient was born.
Q: What was the biggest challenge during development?
A: Filename compatibility on Windows.
Early versions frequently produced garbled characters or failed to create files when downloading FTP resources with Chinese names. The root cause was an encoding mismatch: UTF-8 on FTP servers versus Windows' local ANSI (GBK) encoding.
We implemented dynamic encoding detection and conversion directly in the plugin, which resolved the issue in over 99% of real-world scenarios. It became one of the most appreciated features in user feedback—a small breakthrough, but very satisfying.
Q: Any advice for other developers considering building plugins?
A: If you ever think, "DolphinDB solves most of my problems—but I'm still missing one small tool," then go ahead and build it.
You don't need to aim for something massive. Start with one concrete pain point, like I did with bulk FTP downloads. DolphinDB's documentation and community support are strong, and seeing your plugin save others time is incredibly rewarding.
Q: How would you evaluate DolphinDB as a platform?
A: For me, it's the combination of high performance and low barriers to entry.
On one hand, its columnar storage and parallel computing are perfectly suited for financial big-data workloads. On the other, the plugin framework is extremely developer-friendly: clear C++ structure, seamless integration between scripts and native code, no complex adaptation layers.
The openness of the ecosystem—and the precision of its user community—also stand out compared to more general-purpose platforms.
Q: What's next for ftpClient?
A: Short-term: I'll continue iterating on ftpClient—improving SFTP support, adding transfer progress callbacks and audit logs, and expanding compatibility with niche FTP servers.
Longer-term: I'm watching high-frequency data ingestion needs closely, such as quantitative data decompression and object storage integration (like S3). If real demand emerges, I'd be interested in building new plugins to fill those gaps.
Anyone interested in collaborating or sharing ideas is welcome to reach out:
GitHub: https://github.com/FlossieQiu/ftpClient/issuesEmail: 5512962@qq.com
Interested in the ftpClient plugin? For plugin access, technical details, or collaboration opportunities, please contact us at: i nfo@dolphindb.com
Learn more about us: www.dolphindb.com