1

Introduction

2

My teams main responsibility is to aggregate data from various microservices, and then transform it into larger domain

3

models that other teams are able to consume. As an example, a movie might require metadata, recommendations, the users

4

potential progress, overrides from an editor, and so on. However, for performance reasons, we also need the capability

5

to provide APIs which serves thinner slices of every model.

6

7

In this post, I'll discuss a recent POC I did to try and make this work easier. We'll look at some typescript code that

8

infers types from trees, some trade-offs with directed acyclical graphs, as well as writing tests for an

9

asynchronous traversal. All of the code is available

10

11

API design

12

When working on packages and different types of abstractions, I like to start with code that consumes the API as if it

13

were already written. This helps me avoid sunk cost fallacy – that is, having invested so much time in the internals

14

that I feel forced to find a use-case for it. If the abstraction feels weird or the API is unclear, I'll scrap the idea

15

early.

16

17

Ok, so let's proceed by trying to stub out the API. My purpose with this POC was to develop a method for defining our

18

domain models in such a way that each property could be lazy-loaded as needed, hence enabling us to generate subsets of

19

these models on the fly with just the data you need.

20

21

The initial code I wrote to visualize what the API could look like used a sort of builder pattern like this:

22

23
// getMoviesForUserSubscription combines two models to get the movies that the user is able to watch.
24
async function getMoviesForUserSubscription() {
25
	const subscription = await models.subscription.withUserPackage().compile()
26
	const movies = await models.movie
27
		.withSubscriptionInfo()
28
		.withProgress()
29
		.withRecommendations()
30
		.withPosterImages()
31
		.compile()
32
	return movies.filter((m) => m.subscriptionInfo.subscription.id == subscription.id)
33
}
24

25

Simply pressing . (dot) in the editor, and have the autocomplete outline each model within our domain seemed like an

26

intuitive way to get the data that you needed.

27

28

Next, I had to find a way to construct these models and declare their dependencies. At first, I considered using a

29

directed graph. It was evident that situations could arise where it would be useful to have multiple edges to the same

30

node:

31

32
                 ┌────────┐
33
                 │ Series │
34
                 └────────┘
35
36
      ┌───────────────┘       ┌────────┐
37
      │                       │ Movie  │─────┐
38
      ▼                       └────────┘     │
39
 ┌────────┐                        │         │
40
 │Seasons │──────┐                 │         │
41
 └────────┘      │                 │         │
42
      │          │                 │         ▼
43
      │          │           ┌─────┘    ┌────────┐
44
      │          │           │          │Trailer │
45
      ▼          │           │          └────────┘
46
 ┌────────┐      │           │               │
47
 │Episodes│      │           │               │
48
 └────────┘      │           ▼               │
49
      │          │      ┌────────┐           │
50
      └──────────┴─────▶│Progress│◀──────────┘
51
                        └────────┘
33

34

In the example above, a user could have progress on a series, season, episode, movie and trailer. If we wanted to derive

35

a submodel from that graph it would have to look something like this:

36

37
const episodeWithProgress = await seriesModel.withSeasons().withEpisodes().withProgress().compile()
38
const movieWithProgress = await movieModel.withProgress().compile()
38

39

However, permitting multiple edges to connect to a single node introduces ambiguity in the path to reach it, which would

40

necessitate explicit instructions for the traversal.

41

42

I think this would negatively impact the API because you would either need prior knowledge of the traversal sequence, or

43

would have to rely heavily on your editors autocomplete until you were able to pinpoint the path to the property you

44

wanted.

45

46

We were also not overly concerned with the potential network overhead from retrieving more data than necessary. Using

47

something like GraphQL makes it easy to pick the fields you want with precision, but it can also result in very verbose

48

queries when working with large models. On top of this, it makes the traversal much slower.

49

50

If we instead were to consider the series and movie models as two distinct trees:

51

52
                     ┌────────┐                  ┌────────┐
53
              ┌─────▶│ Series │◀─────┐           │ Movie  │
54
              │      └────────┘      │           └────────┘
55
              │                      │                ▲
56
              │                      │                │
57
         ┌────────┐             ┌────────┐       ┌────────┐
58
     ┌──▶│Seasons │◀──┐         │Progress│       │Progress│
59
     │   └────────┘   │         └────────┘       └────────┘
60
     │                │
61
     │                │
62
┌────────┐       ┌────────┐
63
│Progress│       │Episodes│
64
└────────┘       └────────┘
65
66
67
                 ┌────────┐
68
                 │Progress│
69
                 └────────┘
53

54

We've essentially reversed the direction of the edges so that each node now points to its parent instead. Given there's

55

just one path from any node to the root, we can make the traversal implicit. Users woukd be able to access leaf nodes

56

directly on the model without needing knowledge of the path to resolve it:

57

58
// No need to keep hitting . to try and figure out how a node is
59
// reached. The traversal happens automatically within the library.
60
const episodesWithprogress = await seriesModel.episodesWithprogress().compile()
59

60

This would make the API for consuming the data much nicer, imo. However, treating each model as a distinct tree could

61

result in considerable code duplication.

62

63

At this point, I felt the need to draw out a few more domain models to evaluate the tree structures. After doing so, I

64

concluded that most trees were broad but not very deep.

65

66

Primarily, I wanted to have the capability to segment out costly nodes when the data wasn't necessary:

67

68
                      ┌──────────┐
69
      ┌──────────────▶│  Movie   │◀───────────────┐
70
      │               └──────────┘                │
71
      │                     ▲                     │
72
      │              ┌──────┴──────┐              │
73
      │              │             │              │
74
┌──────────┐   ┌──────────┐  ┌──────────┐   ┌──────────┐
75
│ Progress │   │IsFavorite│  │  Rating  │   │  Images  │
76
└──────────┘   └──────────┘  └──────────┘   └──────────┘
69

70

With a data structure in mind, I also wanted to test the process of breaking out a new node when I had determined that a

71

certain call was costly, and not always essential. Let's say that the progress request below is expensive and difficult

72

to cache:

73

74
import { declareModel } from './model'
75
 
76
async function fetchFullMovie(id: number) {
77
	const metadataRequest = Promise.resolve({ id, title: 'Movie title', rating: 4.5 })
78
	const imagesRequest = Promise.resolve([{ src: 'https://example.com/image.jpg' }])
79
	const progressRequest = Promise.resolve({ watched: 0.5 })
80
	const [metadata, images, progress] = await Promise.all([metadataRequest, imagesRequest, progressRequest])
81
	return { id, metadata, images, progress }
82
}
83
 
84
const movieRootNode = { name: 'movie' as const, run: fetchFullMovie }
85
const movieModel = declareModel(movieRootNode)
86
const movie = await movieModel.compile(1)
75

76

To improve the performance of our system, we'd like the possibility to extract that call to its own node so that it is

77

only fetched when we need it.

78

79

_NOTE:_ For the sake of simplicity, I've created our current tree with a single node. Nonetheless, the procedure for

80

dividing a node and introducing new branches should remain consistent, irrespective of the node's position in within the

81

tree.

82

83

The first thing we would have to do with our current API would be to separate the fetch functions:

84

85
async function fetchMovie(id: number) {
86
	const metadataRequest = Promise.resolve({ title: 'Movie title', rating: 4.5 })
87
	const imagesRequest = Promise.resolve([{ src: 'https://example.com/image.jpg' }])
88
	const [metadata, images] = await Promise.all([metadataRequest, imagesRequest])
89
	return { id, ...metadata, images }
90
}
91
 
92
async function fetchProgress<T extends { id: number }>(obj: T) {
93
	return Promise.resolve({ watched: `${obj.id * 2}%` })
94
}
86

87

We're then able to declare a new child node that we can use to grab the progress:

88

89
const rootNode = { name: 'movie' as const, run: fetchMovie }
90
const progressNode = { name: 'progress' as const, parent: rootNode, run: fetchProgress }
91
const model = declareModel(rootNode, progressNode)
92
 
93
const movieWithoutProgress = await model.compile(5)
94
console.log(movieWithoutProgress.id) // 10
95
 
96
const movieWithProgress = await model.withProgress().compile(10)
97
console.log(movieWithProgress.id) // 10
98
console.log(movieWithProgress.progress.watched) // 20%
90

91

Using the above code, we've constructed the following linear tree:

92

93
        ┌────────┐
94
        │ Movie  │
95
        └────────┘
96
97
98
        ┌────────┐
99
        │Progress│
100
        └────────┘
94

95

This approach might be acceptable if the movie data is heavily cached. Yet, fetching progress does not depend on the

96

movie request. As demonstrated in the fetchFullMovie function, we can retrieve data from the upstream APIs in parallel,

97

as long as we possess the ID.

98

99

This posed a slight challenge for the API. One way to eliminate the dependency between our Movie and Progress nodes

100

could be to introduce a new root node that instantly resolves a promise with the ID. This would make the nodes siblings

101

on the same layer of our tree, thereby enabling both of them to be executed in parallel:

102

103
             ┌────────┐
104
      ┌──────│   Id   │──────┐
105
      │      └────────┘      │
106
      │                      │
107
      │                      │
108
      ▼                      ▼
109
 ┌────────┐             ┌────────┐
110
 │Progress│             │ Movie  │
111
 └────────┘             └────────┘
104

105
const newRootNode = { name: 'id' as const, run: (id: number) => Promise.resolve(id) }
106
const newMetadataNode = { name: 'metadata' as const, parent: newRootNode, run: fetchMovie }
107
const newProgressNode = { name: 'progress' as const, parent: newRootNode, run: fetchProgress }
108
const model = declareModel(newRootNode, newMetadataNode, newProgressNode)
109
 
110
// Create a model with just the progress
111
const onlyProgress = await model.withProgress().compile(5)
112
console.log(onlyProgress.progress.watched)
113
 
114
// Create a model with just the metadata
115
const onlyMetadata = await model.withMetadata().compile(10)
116
console.log(onlyMetadata.metadata.rating)
117
 
118
// Create a full model with all data
119
const fullMovie = await model.withMetadata().withProgress().compile(15)
120
console.log(fullMovie.progress.watched)
121
console.log(fullMovie.metadata.rating)
122
console.log(onlyMetadata.metadata.rating)
106

107

Looking at the code above, we can see that the process of attaching a node is simple. All we have to do is specify the

108

parent node we depend on and implement a 'run' function. This function receives the value that the parent resolves to.

109

The new node can then leverage this data for its own fetching.

110

111

The final aspect I wanted to prototype before diving into the implementation was to see explore how it would look like

112

if we wanted to "share" nodes between different trees. Consider this diagram where every media type could have progress:

113

114
                     ┌────────┐                  ┌────────┐
115
              ┌─────▶│ Series │◀─────┐           │ Movie  │
116
              │      └────────┘      │           └────────┘
117
              │                      │                ▲
118
              │                      │                │
119
         ┌────────┐             ┌────────┐       ┌────────┐
120
     ┌──▶│Seasons │◀──┐         │Progress│       │Progress│
121
     │   └────────┘   │         └────────┘       └────────┘
122
     │                │
123
     │                │
124
┌────────┐       ┌────────┐
125
│Progress│       │Episodes│
126
└────────┘       └────────┘
127
128
129
                 ┌────────┐
130
                 │Progress│
131
                 └────────┘
115

116

Having to declare multiple nodes like this could lead to substantial code duplication, thereby making the models harder

117

to maintain.

118

119

Therefore, I chose to prototype a simplified version of the two trees above, where both series and movies could have

120

progress appended to them. The progress node, however, is solely concerned with IDs, and its endpoint can fetch progress

121

for any media type. Let's see what that might look like:

122

123
function fetchSeries(id: number) {
124
	return Promise.resolve({
125
		id,
126
		name: 'name of the series',
127
		seasonIds: ['1', '2', '3'],
128
	})
129
}
130
 
131
function fetchMovie(id: number) {
132
	return Promise.resolve({
133
		id,
134
		name: 'name of the movie',
135
	})
136
}
137
 
138
function fetchProgress<T extends { id: number }>(obj: T) {
139
	return Promise.resolve({ percentageWatched: `${obj.id * 2}%` })
140
}
141
 
142
const seriesRoot = { name: 'series' as const, run: fetchSeries }
143
const seriesProgress = { name: 'progress' as const, parent: seriesRoot, run: fetchProgress }
144
 
145
const movieRoot = { name: 'movie' as const, run: fetchMovie }
146
const movieProgress = { name: 'progress' as const, parent: movieRoot, run: fetchProgress }
124

125

The code above creates two distinct linear trees. Yet, the only part that gets duplicated between the two trees, are

126

these two lines:

127

128
const seriesProgress = { name: 'progress' as const, parent: seriesRoot, run: fetchProgress }
129
const movieProgress = { name: 'progress' as const, parent: movieRoot, run: fetchProgress }
129

130

By using generic fetch functions, we can conceptually think of the two trees above as a directed acyclic graph as they

131

are able to share code for fetching and transforming the data:

132

133
┌────────┐       ┌────────┐
134
│ Movie  │       │ Series │
135
└────────┘       └────────┘
136
     │                │
137
     │                │
138
     │                │
139
     │                │
140
     │   ┌────────┐   │
141
     └──▶│Progress│◀──┘
142
         └────────┘
134

135
const seriesModel = createModel(seriesRoot, seriesProgress)
136
const movieModel = createModel(movieRoot, movieProgress)
137
 
138
const series = await seriesModel.withProgress().compile(14)
139
console.log(series.progress.percentageWatched) // 28%
140
 
141
const movie = await movieModel.withProgress().compile(8)
142
console.log(movie.progress.percentageWatched) // 16%
136

137

Adding types

138

For the models to be useful, they must be able to accurately infer the types. Let's reuse some of the code from earlier:

139

140
const rootNode = { name: 'id' as const, run: (id: number) => Promise.resolve(id) }
141
const metadataNode = { name: 'metadata' as const, parent: rootNode, run: fetchMovie }
142
const progressNode = { name: 'progress' as const, parent: rootNode, run: fetchProgress }
143
const model = createModel(rootNode, metadataNode, progressNode)
141

142

In the example above, the model should only contain functions for selecting metadata and progress. Since tree traversal

143

always starts from the top, the rootNode is considered the base of our model. Therefore, we'll add the values it

144

returns to the root of the submodel we attain by calling compile later.

145

146

Consequently, I anticipate the model in the above example to include the following three functions:

147

148
model.withMetadata()
149
model.withProgress()
150
model.compile()
149

150

The functions responsible for lazy-loading the metadata and progress should take the name of the node and append the

151

word with. Additionally, I want to ensure the name of the root node is excluded. If included, it should trigger a type

152

error:

153

154
model.withId() // Error: withId() does not exist on type { ... }
155

156

For the compile call's return value, I expect the return type to be the intersection of the root node and the return

157

values from all withX() invocations:

158

159
// { id }
160
const movieOne = await model.compile()
161
 
162
//  { id, progress: { ... } }
163
const movieTwo = await model.withProgress().compile()
164
 
165
//  { id, metadata: { ... } }
166
const movieThree = await model.withMetadata().compile()
167
 
168
//  { id, metadata: { ... }, progress: { ... } }
169
const movieFour = await model.withMetadata().withProgress().compile()
160

161

To add achieve these types we need to traverse the tree, and add functions for accessing each node. The compile function

162

must keep track of the functions we've invoked to ensure it returns the appropriate type:

163

164
type Func<TIn = any, TOut = any> = (arg: TIn) => Promise<TOut>
165
 
166
interface RootNode<N extends string = string, TIn = any, TOut = any> {
167
	name: N
168
	run: Func<TIn, TOut>
169
}
170
 
171
interface ChildNode<N extends string = string, TIn = any, TOut = any> extends RootNode<N, TIn, TOut> {
172
	parent: TreeNode
173
}
174
 
175
type TreeNode<N extends string = string, TIn = any, TOut = any> = RootNode<N, TIn, TOut> | ChildNode<N, TIn, TOut>
176
 
177
type IsChildNode<T> = 'parent' extends keyof T ? true : false
178
 
179
type EnsureNodeArray<T> = T extends TreeNode[] ? T : never
180
 
181
type FindRootNode<T extends TreeNode[]> = T extends [infer First, ...infer Rest]
182
	? IsChildNode<First> extends true
183
		? FindRootNode<EnsureNodeArray<Rest>>
184
		: First
185
	: never
186
 
187
type ExtractNode<M extends TreeNode[], N extends string> = Extract<M[number], { name: N }>
188
 
189
type NodeName<M> = M extends TreeNode<infer N>[] ? N : never
190
 
191
type RemoveFuncPrefix<T extends string> = T extends `with${infer U}` ? Uncapitalize<U> : never
192
type AddFuncPrefix<T extends string> = `with${Capitalize<T>}`
193
 
194
type Model<T, M extends TreeNode[]> = Omit<
195
	{
196
		[K in AddFuncPrefix<NodeName<M>>]: K extends string
197
			? () => Model<T & Record<RemoveFuncPrefix<K>, Awaited<ReturnType<ExtractNode<M, RemoveFuncPrefix<K>>['run']>>>, M>
198
			: never
199
	},
200
	FindRootNode<M> extends { name: string } ? AddFuncPrefix<FindRootNode<M>['name']> : never
201
> & {
202
	compile: (
203
		...args: FindRootNode<M> extends { run: (...args: any) => any } ? Parameters<FindRootNode<M>['run']> : never
204
	) => Promise<T> &
205
		(FindRootNode<M> extends { run: (...args: any) => any } ? ReturnType<FindRootNode<M>['run']> : never)
206
}
207
 
208
export function declareModel<T, M extends TreeNode<any, any, any>[]>(...nodes: M): Model<T, M> {
209
	// ...
210
}
165

166

With the types above I could inspecting the code I used earlier to design the API. The types were being inferred

167

correctly, which meant that it was time to move on to the actual implementation.

168

169

Implementation

170

When working on POCs, where the code may be discarded, I still think it's important to ensure that the external API is

171

accurately typed. However, when the types are this complex, I sometimes use any on the internal functions to be able

172

to quickly iterate on the implementation using "vanilla" javascript.

173

174

My vision for the internal workings of the builder was to resolve the tree layer-by-layer. Nodes within the same layer

175

of the tree could be executed in parallel, and their children would follow in the next cycle, and so on. This approach

176

felt well-suited for the wide, shallow trees I've described before.

177

178

Therefore, the traversal is always going to start at the root - making a function to identify the root node a logical

179

first step. Since our current types neither mandate a single root node, nor limit its number, I included two runtime

180

checks:

181

182
function extractRootNode<M extends TreeNode[]>(nodes: M) {
183
	const rootNodes = nodes.filter((x) => !('parent' in x))
184
 
185
	if (rootNodes.length === 0) {
186
		throw new Error('No root node found')
187
	}
188
 
189
	if (rootNodes.length > 1) {
190
		throw new Error('You can only have one root node')
191
	}
192
 
193
	return rootNodes[0] as RootNode
194
}
183

184

This is something I could revisit and enforce with types if the POC was a success. Now, before I could think of the

185

traversal, I had to use the function above to identify the root node, and maintain a record of all nodes that I wanted

186

to visit:

187

188
export function declareModel<T, M extends TreeNode<any, any, any>[]>(...nodes: M): Model<T, M> {
189
	const rootNode = extractRootNode(nodes)
190
	const parentChildrenToVisit = new Map<TreeNode, Set<TreeNode>>()
191
	// ...
192
}
189

190

Next, I created a builder object and attached functions dedicated to visiting each of the nodes. Keep in mind that

191

visiting the root node is implicit, so it won't be part the API. Therefore, we'll only add nodes with a parent:

192

193
export function declareModel<T, M extends TreeNode<any, any, any>[]>(...nodes: M): Model<T, M> {
194
	// ...
195
	const builder: any = {}
196
 
197
	// Add functions that, when invoked, marks a node for visitation
198
	nodes.forEach((node) => {
199
		if ('parent' in node) {
200
			builder[addFunctionPrefix(node.name)] = function () {
201
				addNodeToVisit(node)
202
				return builder
203
			}
204
		}
205
	})
206
}
194

195

By looking at the code above, we can see that the function names for the builder are determined by another function

196

called addFunctionPrefix. Its role is to add with as a prefix to the node name, and ensure that the first letter of

197

the name is capitalized:

198

199
function addFunctionPrefix(name: string) {
200
	return `with${name.charAt(0).toUpperCase() + name.slice(1)}`
201
}
200

201

Recall that I aimed for the API to allow users to access any node, regardless of its position in the tree. Now suppose

202

we add this node, which is located five layers below the root, to our map.

203

204
const x = await model.withNodeFiveLayersDown().compile()
205

206

There would be no way for us to reach that node directly from the root. Hence, we need to ensure that with each node we

207

want to visit we also request a visit to its parent, all the way to the top:

208

209
export function declareModel<T, M extends TreeNode<any, any, any>[]>(...nodes: M): Model<T, M> {
210
  // ...
211
 
212
  const addNodeToVisit = (node: M[number]) => {
213
    // Break the recursion if we've reached the root.
214
    if (!("parent" in node)) {
215
      return
216
    }
217
 
218
    // Register that we want to visit this node after our parent node has been processed.
219
    // Remember, a sibling node might already have added itself to the set.
220
    const siblings = parentChildrenToVisit.get(node.parent)
221
    if (siblings) {
222
      siblings.add(node)
223
    } else {
224
      parentChildrenToVisit.set(node.parent, new Set([node]))
225
    }
226
 
227
    // This could be a leaf node with no direct connection to the root.
228
    // Therefore, we need to walk up its ancestry path until we reach the
229
    // top, which is going to serve as the starting point of the traversal.
230
    addNodeToVisit(node.parent)
231
  }
232
 
233
    // ...
210

211

With the capability to attach functions to the builder, and register nodes for visitation, we can move on to the compile

212

function, which will handle the actual traversal:

213

214
export function declareModel<T, M extends TreeNode<any, any, any>[]>(...nodes: M): Model<T, M> {
215
  // ...
216
 
217
  builder.compile = async (...args: any[]) => {
218
    // Base the model on the return type of the root node.
219
    let model = await (rootNode as any).run(...args)
220
    // Exit early if we don't have any child nodes to visit.
221
    if (parentChildrenToVisit.size === 0) {
222
      return model
223
    }
224
 
225
    // We'll slice the tree layer by layer. Here, we'll start with
226
    // nodes that are direct descendants of the root node.
227
    const children = parentChildrenToVisit.get(rootNode)
228
    // The queue will contain tuples: the first value represents
229
    // the node to execute, and the second is the resolved value
230
    // from its parent node, used as input for the "run" function.
231
    const queue = [...(children ?? [])].map((c) => [c, model])
232
 
233
    // Continuiously add properties to the model with each layer of nodes.
234
    while (queue.length > 0) {
235
      const nodesWithArgs = queue.splice(0, queue.length)
236
      // These nodes carry their required input in the tuple and can run concurrently.
237
      const promises = nodesWithArgs.map(([node, arg]) => node.run(arg))
238
      const responses = await Promise.all(promises)
239
 
240
      // Decorate the model with fields from this layer
241
      model = responses.reduce((acc, cur, i) => ({ ...acc, [nodesWithArgs[i][0].name]: cur }), model)
242
 
243
      // Loop through the nodes to see if we have any child nodes marked
244
      // for visitation. If found, we add them to the queue along with
245
      // the node's resolved value, which serves as their input.
246
      for (let i = 0; i < nodesWithArgs.length; i++) {
247
        const childrenToVisit = parentChildrenToVisit.get(nodesWithArgs[i][0])
248
        if (!childrenToVisit) {
249
          continue
250
        }
251
        queue.push(...[...childrenToVisit].map((x) => [x, responses[i]]))
252
      }
253
    }
254
    return model
255
  }
256
 
257
  return builder
215

216

The traversal logic is fairly simple. We maintain a queue holding tuples of nodes and the resolved values from the

217

preceding layer. The loop continues until the queue is empty. During each cycle, all nodes in the queue are executed

218

concurrently. If any of their children are registered for a visit, they are added to the queue.

219

220

At this point I was able to execute the code from the previous examples, and could verify that they yielded the results

221

I expected. The next step would be to write tests to ensure that the traversal was efficient, and that no node was being

222

called more than once.

223

224

Testing

225

I began by creating a function that would return eight nodes that formed the following tree:

226

227
 
228
                    ┌─────┐
229
           ┌───────▶│  0  │◀───────┐
230
           │        └─────┘        │
231
           │                       │
232
           │                       │
233
           │                       │
234
           │                       │
235
        ┌─────┐                 ┌─────┐
236
   ┌───▶│  1  │◀──┐             │  2  │
237
   │    └─────┘   │             └─────┘
238
   │              │                ▲
239
   │              │                │
240
┌─────┐        ┌─────┐          ┌─────┐
241
│  3  │        │  4  │     ┌───▶│  5  │◀───┐
242
└─────┘        └─────┘     │    └─────┘    │
243
                           │               │
244
                           │               │
245
                        ┌─────┐         ┌─────┐
246
                        │  6  │         │  7  │
247
                        └─────┘         └─────┘
248
 
228

229
function createNodes() {
230
	const nodeZero = { name: 'zero' as const, run: (x: number) => resolveValue(x) }
231
	const nodeOne = { name: 'one' as const, parent: nodeZero, run: (_: number) => resolveValue(1) }
232
	const nodeTwo = { name: 'two' as const, parent: nodeZero, run: (_: number) => resolveValue(2) }
233
	const nodeThree = { name: 'three' as const, parent: nodeOne, run: (_: number) => resolveValue(3) }
234
	const nodeFour = { name: 'four' as const, parent: nodeOne, run: (_: number) => resolveValue(4) }
235
	const nodeFive = { name: 'five' as const, parent: nodeTwo, run: (_: number) => resolveValue(5) }
236
	const nodeSix = { name: 'six' as const, parent: nodeFive, run: (_: number) => resolveValue(6) }
237
	const nodeSeven = { name: 'seven' as const, parent: nodeFive, run: (_: number) => resolveValue(7) }
238
	return [nodeZero, nodeOne, nodeTwo, nodeThree, nodeFour, nodeFive, nodeSix, nodeSeven] as const
239
}
230

231

I think the size of this tree is sufficient for testing one of the trickier use cases, where we access just two leaf

232

nodes, say node 3 and 7.

233

234

Subsequently, I added the resolveValue function that you can see above. The implementation is very straightforward; it

235

wraps any given value in a promise that resolves immediately using setTimeout:

236

237
function resolveValue(value: number): Promise<{ value: number }> {
238
	return new Promise((resolve) => {
239
		setTimeout(() => resolve({ value }), 0)
240
	})
241
}
238

239

This allows me to use fake timers, and advance time to have the callbacks executed. I also added a function for flushing

240

the promises for the preceding layer, and another to ensure that each node in a given layer is called only once, and

241

with the correct arguments:

242

243
// Used to flush out the promises from the layer above
244
async function flushNumOfParentNodes(numberOfNodes: number) {
245
	for (let i = 0; i < numberOfNodes; i++) {
246
		await Promise.resolve()
247
	}
248
}
249
 
250
function assertNodeInvocations(nodes: ReturnType<typeof createNodes>, indexArgs: Array<[number, number]>) {
251
	nodes.forEach((node, i) => {
252
		const idxArg = indexArgs.find(([index]) => index === i)
253
		if (idxArg) {
254
			expect(node.run).toBeCalledTimes(1)
255
			expect(node.run).toBeCalledWith(idxArg[0] == 0 ? idxArg[1] : { value: idxArg[1] })
256
		} else {
257
			expect(node.run).toBeCalledTimes(0)
258
		}
259
	})
260
}
244

245

With that, I could write an actual test:

246

247
describe('declareModel', () => {
248
	it('processes the tree layer by layer into a model', async () => {
249
		const nodes = createNodes()
250
		const modelBuilder = declareModel(...nodes)
251
 
252
		// Enable fake timers. We'll want to assert that the tree is processed layer
253
		// by layer. We'll also attach spies to the run function of every node.
254
		jest.useFakeTimers()
255
		nodes.forEach((node) => jest.spyOn(node, 'run'))
256
 
257
		// We are going to select a subset from the entire domain model by
258
		// picking node 3 and 7. Initially, we won't await this call. This
259
		// will enable us to to make assertions on the traversal.
260
		const modelPromise = modelBuilder.withThree().withSeven().compile(20)
261
 
262
		// Layer 1 should only call the run function of the root
263
		// node with the value passed to the compile function.
264
		assertNodeInvocations(nodes, [[0, 20]])
265
 
266
		// Proceed to layer 2 by advancing time and flushing the root node promise.
267
		jest.advanceTimersByTime(0)
268
		await flushNumOfParentNodes(1)
269
 
270
		// Layer 2 is expected to invoke the 'run' function for nodes 1 and 2, passing
271
		// in the values that have been resolved from each node's direct parent.
272
		assertNodeInvocations(nodes, [
273
			[0, 20],
274
			[1, 20],
275
			[2, 20],
276
		])
277
 
278
		// Proceed to layer 3 by advancing time and flushing the promises from layer 2.
279
		jest.advanceTimersByTime(0)
280
		await flushNumOfParentNodes(2)
281
 
282
		// At this point, we're experiencing diminishing returns from continuously
283
		// asserting that each subsequent layer wasn't called. However, this is a
284
		// fairly small tree, so I think the extra assertions are justified.
285
		assertNodeInvocations(nodes, [
286
			[0, 20],
287
			[1, 20],
288
			[2, 20],
289
			[3, 1],
290
			[5, 2],
291
		])
292
 
293
		// Proceed to layer 4 by advancing time and flushing promises.
294
		jest.advanceTimersByTime(0)
295
		await flushNumOfParentNodes(3)
296
 
297
		// Now we should have invoked the run method of node 7 which is our last node.
298
		assertNodeInvocations(nodes, [
299
			[0, 20],
300
			[1, 20],
301
			[2, 20],
302
			[3, 1],
303
			[5, 2],
304
			[7, 5],
305
		])
306
 
307
		// Advancing one more time here will allow the modelPromise to resolve.
308
		jest.advanceTimersByTime(0)
309
 
310
		// We can now await the modelPromise and assert the structure.
311
		const model = await modelPromise
312
 
313
		// The root value is going to be whatever we used to start the tree traversal.
314
		expect(model.value).toBe(20)
315
		expect(model.three.value).toBe(3)
316
		expect(model.seven.value).toBe(7)
317
	})
318
})
248

249

With the test passing, we've confirmed that the nodes are invoked in the correct order, the appropriate number of times,

250

and with the correct arguments. Additionally, we've asserted that nodes on the same layer are executed in parallel

251

during the same tick.

251

252

The end

253

I usually tweet something when I've finished writing a new post. You can find me on Twitter

254

by clicking 

normalintroduction.md
||833:30

Recently Edited

Recently Edited

File name

Tags

Time to read

Created at

context

  • go
  • context
8 minutes2024-02-28

circular-buffers

  • go
  • concurrency
  • data processing
5 minutes2024-02-04

go-directives

  • go
  • compiler
  • performance
4 minutes2023-10-21

async-tree-traversals

  • node
  • trees
  • graphs
  • typescript
19 minutes2023-09-10

All Files

All Files

  • go

    5 files

  • node

    2 files

  • typescript

    1 file

  • frontend

    1 file

  • workflow

    7 files