Tree2Tree started as an exploration of the structural performance of L-systems using computer vision. We got some promising results and we have continued the endeavor to discover and predict L-systems behavior with the methods of GAN.

L-systems – tree structures

L-system or Lindenmayer system is a parallel rewriting system and a type of formal grammar. L-system consists of an alphabet of symbols that can be used to make strings, a collection of production rules that expand each symbol into some larger string of symbols, an initial “axiom” string from which to begin construction, and a mechanism for translating the generated strings into geometric structures. L-systems were introduced and developed in 1968 by Aristid Lindenmayer, a Hungarian theoretical biologist, and botanist at the University of Utrecht. Lindenmayer used L-systems to describe the behavior of plant cells and to model the growth processes of plant development. L-systems have also been used to model the morphology of a variety of organisms and can be used to generate self-similar fractals.

Method – pix2pix

Pix2Pix is a Generative Adversarial Network, or GAN, model designed for general purpose image-to-image translation. The approach was presented by Phillip Isola, et al. in their 2016 paper titled “Image-to-Image Translation with Conditional Adversarial Networks” and presented at CVPR in 2017.

Problem

Predict the structural analytical model from a more detailed tree view. A real-life example of this could be to predict the analytical model of a tree as an image based on a tree photo. To train the pix2pix model we required an L-systems dataset with both realistic and structural analytical visualizations and/or images of trees. Since we were not able to find a dataset that has this kind of data we created our own script to generate the dataset.

Generative dataset

Through generative L-systems Grasshopper script that we have developed we have managed to generate 980 realistic tree visualizations and the same amount of structural analytical tree visualizations.

 

Structural view

Realistic view

AI use-case

The use of AI in the current case of predicting structural analytical images based on other representations can have multiple uses. Starting from the fact that L-systems are notoriously difficult to predict in terms of structural behaviors. The analytical images can be used to analyze structures and be tied into further processes for detecting structural performance and/or defects.

A simple example could be to collect photos from parks and based on them see if there might be any branches of trees that might be structurally unstable.

Another example would be to use the analytical image to recreate an actual parametric model. Pix2pix gives us limited possibilities to include metadata as labels, though we have researched other methods that we would like to explore.

In this way, we would be able to create a process that rebuilds from images parametric structural L-systems models. So by taking photos of trees, we can get their structural analytical models in Grasshopper.

Trial 1

Architecture: Epochs 20 / Learning Rate 0.002 / Direction A To B / Batch Size 1 / Training Dataset 500

discrim_loss 0.6851692

gen_loss_GAN 2.2308598

gen_loss_L1 0.025927672

HYPOTHESIS: The initial test was to check that there were no problems with the dataset that we created and the model was working. It also gave us an initial set of parameters to work with and set a basis for other tests.

RESULTS: Visually the outputs look interesting but there are areas where the strokes are not visible or very faint.

Trial 2

Architecture: Epochs 5 / Learning Rate 0.002 / Direction B To A/ Batch Size 1 / Training Dataset 500

discrim_loss 1.0663649

gen_loss_GAN 1.2199634

gen_loss_L1 0.044370756

HYPOTHESIS: Moving on to B to A for 5 epochs to test the visual output as well as the stability of the model. All other parameters remain the same in this test.

RESULTS: The model seems more stable than the previous test and the visual results are not well defined in certain areas such as the edges. This can be improved by either increasing the epoch or batch size.

Trial 3

Architecture: Epochs 25 / Learning Rate 0.002 / Direction B To A / Batch Size 1 / Training Dataset 500

discrim_loss 0.7276931

gen_loss_GAN 1.9181364

gen_loss_L1 0.036683336

HYPOTHESIS: Will the output results improve as a result of increasing the maximum no. of epoch. Will this affect the generator and discriminator loss?

RESULTS: The outputs have improved, and the images are sharper around the edges and the shadows are better defined as well. The third image looks almost identical to the input.

Next steps

The next steps would be to have more complex images, perhaps a set of trees with more variations in color, tone size, and the type and slowly moving to a natural landscape setting like a jungle or park.

Conclusion

Being a simple image, it did not take a lot of epochs to train the model to a stage where the output was showing good results. The predictions for both A to B and B to A look more realistic (like real branches) than the input images, which were straight lines. We felt this was quite an interesting outcome. Although the branches and trees here were digitally generated it would be worth trying them with images of real trees and branches and comparing the results.

Tree2Tree is a project of IAAC, Institute for Advanced Architecture of Catalonia developed in the Generative building design 2020/2021 by Students: Amar Gurung & Polina Hadjimitova and Faculty Stanislas Chaillou and faculty assistant Oana Taut.