The PJHoon

Initially I built PJHoon with 2 goals: consolidate my technical knowledge, and create a, part of, so-called “legacy”. This project is the most passionate one and most hand-crafted one (because AI had not been that productive and smart yet). I coded from line to line, from Frontend Angular, to Backend Spring, to Database Postgres/ClickHouse, to Infrastructure Kubernetes. I paid attention to the very minor details like the animations, the element placements, the responsiveness, the API names, method arguments counts, the indices, the modelings, partition schemes, the queries, the deployment manifests, the HelmReleases values, … It was fun (even though it was also exhausting), and for a long time since my mentor left the company did I feel like learning something interesting and insightful again. I did achieve one goal, excellently.

The Data Platform

The data platform has several properties:

• It is complex
• It serves in large scale
• It integrates with other systems, a lot
• Working with data is not just about performance and freshness, it’s about security (and credit) as well.

The last 3 points was slowly sapping my spirit. I, myself, would have never produced close to 1 billion records for the “P99 sub-second query” SLA I made when building this.

When I was working in PrimeData, as the Tech Lead of the Integration Team, it is obvious to me that doing Integrations (especially building connectors that pull external data) is THE MOST dynamic workload in the system. We dealt with poor API scheme, vendor’s bugs, vendor’s data freshness, lack of incremental-sync supports, … sometimes, it’s just that the vendor won’t expose the API to keep their user/data base’s advantage. Others tried crawling, and it is even more painful navigating through HTMLs and working around Bot prevention schemes from vendors. More importantly, these works are more often than not specific to a vendor, business-context heavy, and rarely reusable across clients. Yet, absurdly, Integration connectors are usually of the most valuable components in a Data Platform. To make the most of the data platform, it should have as much data as possible for several use cases. And, well, guess where data comes from.

And then I self-raised a question: what about security, even with encryption in-transit and at-rest, would a company trust a “Cloud” Data Platform startup from a single-digit-years-of-experience Data Engineer instead of GCP? If we setup on-premise, what is our selling-point, what is our asset?

The Archival - Temporarily

I slowly realize Data Platform is, by nature, internal to a company. A segment with wider audience (but narrower use case) would be a monitoring platform, which is already market-saturated. There are little feasible use cases for using PJHoon even though it is wildly powerful, so I archived it. But it definitely won’t die, I will resume it one day when we are more mature, in an era where data is crucial asset to any entity. It will still be there to remind me of the time I had fun with coding and probing and messing around with both hardware and software, which is gradually stolen by AI.