I tend to talk in black and white quite a bit but people who know me realize when certain use-cases do not fit, even I break from my black or white view. ie. Take everything here with a grain of salt and feel free to challenge it.
Let's get real here, it doesn't exist but I certainly came close. My hobby project is in-use in production and definitely leads to more productive teams and is even inches away from test case generation. My goal for a development environment is two-fold in 'my ideal world' and in this order.
Team Productivity
Individual Productivity
Now, I have a whole presentation on team speed but I am going to focus more on development environment in this blog. There is a whole list of choices for an environment to be addressed in a startup that you will have. Ideally, you will have some 5 year plan (and working backwards, a 3 year and 1 year plan as well). Here are just some of those choices
Language Choice - typesafe or not
example: typescript vs. javascript
Monolith vs. microservices vs. an even better path
Monorepo or many repos
(if monorepo) - monobuild or many builds on monorepo
Testing - Feature Testing or Unit Testing
Ability to override production code for testing
Automatically integrate database changes(DDL & DML) for local dev environment
Local Debugging
Ability to override production code for local dev
Hot recompile if typesafe so no server restarts
There is too much info in my head on this topic so I am going to focus on the Team Productivity aspect and leave many of the details around individual speed to another article.
Programming Language
First, from a language point of view, a typesafe language can reduce bugs by 15% yielding a more productive environment. The other option is a language that is not typesafe 'but' you can run a typechecker like python3. You also need to worry about hiring though as well and developers are enticed by cool technology and there are two paths to that
cool new language
cool technology in the language
We chose an old school language(java) so went with #2 here, much more advanced technology. Developers love a productive environment. Picking a language for a startup is critical for hiring and you should check out the statistics on github most used languages over time to help you there (and probably pick a type-safe language over one that can be typechecked). Unfortunately for ML/AI, you are most likely going to have to pick python so you would want to use the typechecker for python. You will hear many developers tell business people "We need to pick the right language for the job". In reality, you are only going to get maybe a 10% boost from that one developer(or the few that know that language). You will easily lose that boost or worse when hiring and trying to find people. Let's respond to that quote with
Perfect is the enemy of good enough
Instead, go with a mainstream language from github or one that looks like it is becoming mainstream. I have a preference for java, but honestly, I really like the idea behind typescript being used in mobile(react-native), web(react), backend(nodejs) and being able to work in all 3 tiers without switching languages. Hiring has been a mixed-bag in typescript as it is like hiring in China where the ratio of good to bad developers is 1 to 100 instead of 1 to 10 so it is harder to find talent. Tapping into the java developers, it was a bit of a challenge to find java developers that wanted to become typescript developers as well.
Monorepo or many repos
Since I address monolith and microservices in another article, I will just give a quick summary here. IMHO, start with a 'monolith that scales to microservices' and switch to microservces after you find product fit(except for special exceptional cases). If you are a monolith, I am all for creating a repository called monorepo and starting with these directories
monorepo/monolith
monorepo/services/* - to be used years later
monorepo/libraries/internal_apis - to be used years later
monorepo/libraries/external_apis - to be used years later
monorepo/libraries/util - to be used years later
My perfect development environment is definitely microservices even though I do not start with microservices. There is too many advantages to microservices and plenty of blog articles on that topic already. Assuming microservices, let's move on.
Development Environment for Team Speed
First, let's assume quality is non-negotiable and must meet a certain bar.. Let's also assume our cost will be fixed in the glorious project management triangle. Finally, let's also assume, we are not going to change scope for this 'experiment'. In reality none of that is true on a real project but start with theory and extrapolate into reality.
We now want to do more in the fixed time than we could before. We have found we can in many cases double the team speed and keep quality. Here is how we attack that
Feature Testing and throw away 99% of unit tests
Monobuild and breaking before landing on master
80% of Smoke Tests can run single threaded pre-master
Auto-integration of DDL/DML changes on team
Feature Test case generation
On #1, one example was a major swap of dataflow/apache-beam as we were having a ton of production issues. We ran the same exact test suite after the switch was made before changes landed on master via the monobuild. This was done in 2 weeks with 2 developers and no customers reported any bugs when finished. We did very little QA and our company actually had zero QA people. In general, major refactorings on a microservice can result in savings of 30k to 150k per major refactor using feature testing.
On #2, when a developer changes a microservice, our CI monobuild times are under 3 minutes. When someone changes a core library that every microservice depends on, the CI build time can be around 15 minutes. It is quite resonable. The key setup however is the client and server relationship in our monorepo/monobuild. We have some api like AuthApi in our monorepo/libraries/internal-apis/auth-api. We have a microservice with a controller that implements that api like so
public class AuthController implements AuthApi {..... }
We then have the client microservice use the same api. If any developer changes that api, then the monobuild will see the client and the server both depending on the auth-api and will build all 3 things in this order
auth-api
server and client (order doesn't matter for these two)
In so doing, we are catching developers making mistakes and forcing them into changing both sides of the contract. Please do realize, there is some training in teaching teams to 'add-only'. The steps for a correct api refactor are
add something to contract
implement it in server
deprecate old thing but leave it
put changes in production so you it is not in use yet
prelim testing
put code in so you can slowly redirect clients off of the old to the new
put changes in production
ramp production traffic to the new endpoint(or rollback traffic to old if run into issues)
after production traffic is 100% on new added feature, remove old deprecated thing
Then, there is #3, running some smoke test before it lands on master instead of a preproduction environment. I have only done this twice in my career. Once was at Twitter, and the next time was at Orderly with 4 microservices end to end. This is the holy grail and is possible. The correct design for testability is quite critical here and many platforms, you cannot be single threaded.
Another thing affecting team speed is the integration of DDL and putting data into the local development environment for testing purposes. For this, we use an in-memory database and allow developers to modify hibernate entities and seamlessly add data so all conflicts are resolved when merging with each other via a PopulateDatabase.java class that runs when you startup the server locally. We also use a conflict driven method for putting it in pre-production as well so developers can integrate with other developers on merge. Of course, do not forget to have a rollback strategy as well.
Finally is the holy grail which could be more individual speed except test generally improve team speed so developers are not breaking each other's features. This is feature test case generation which is nearly complete in webpieces. The recording piece is done and all that is left is spitting out the test case via logs. If we assume testing code should be 50% or even more of the code base, then generating test can help to significantly reduce the code we write while maintaining the quality.
Summary
Unfortunately, I have not gotten into the details yet around Individual speed with code examples. I will do that in part 2 of this blog. For now, we will leave this with just a focus on team speed and how we can help the team not break each other. One really nice quality of this environment is your best developers will constantly be shipping code while the worst developers will actually struggle to ship anything as they constantly get stuck in trying to get tests to pass. To help the lower tier developers to be more productive, you will actually need the team to start to instill very common patterns like the one we used on a few teams at twitter.