Artificial Toolpath // S.3 SOFTWARE
The project Artificial Toolpath continues the second term’s MRAC Studio project Sensing Sublime and its focus on the use of 3D-Scans as a tool beyond surveying. The goal is to digitize Complex Topologies of 3D-Scans into depth maps and train a neural net to output a robot toolpath for the fabrication of concrete molds.
Motivation
Based on the presence of this iconic architectural heritage within Barcelona, we were interested in the enhancement of intricate surfaces and the value they can bring onto a building’s facades.
Figure 1: Complex Topologies in the example of Catalan Modernism
Figure 2: Pattern and tool marks from several chisel techniques of stone masonry
Context
Within the second term’s Studio project we were able to capture point clouds of iconic facades within the city of Barcelona. The final outcome was the use of sections extracted from the point clouds and use these sections for lofted iterations informed by Barcelona’s iconic architects of Catalan Modernism.
Figure 3: Data Acquisition – Capturing Pointclouds | Source: Sensing Sublime, IAAC
The next step was to use the 3D-scans in a less abstract manner by training a neural net to be able to output a robot toolpath for a concrete casting process from random 3D-scans.
Figure 4: Complex Topology – Iconic built facade features | Source: Sensing Sublime, IAAC
Toolpath
From 3D-Scan – to segmented facade panel – to robot toolpath. During the third term of the Studio seminar, the project Sandcast developed a robot setup for rapid fabrication of customized concrete panels.
Figure 5: Robotic Setup – Studio Project Term III | Source: Sandcast, IAAC
The shape of the panels is based on the 3D-scans captured from iconic buildings within the city of Barcelona. The toolpath is generated from the extraction of the surface isocurves.
Figure 6: Robot Simulation – Sandbox Setup | Source: Sandcast, IAAC
The aim of the Sandcast project is to develop an affordable and fast way to prepare molds for concrete casting. The modulation of sand with the use of a robotic arm and the extraction of the sand with a vacuum is a fast and reusable alternative to concrete molds from more common materials used by the industry like wood.
Figure 7: Negative Form – Sand mold for concrete casting | Source: Sandcast, IAAC
The visibility of the robot toolpath and the generated flow pattern from the robot’s end-effector is a welcome side effect. A more refined, but ‘flat’ surface is achievable but not necessarily in favour.
Figure 8: Concrete Texture – Complex Cast | Source: Sandcast, IAAC
The pattern generated by the robot reminds almost of hand-crafted surfaces. The tooled patterns from a chisel by a stonemason.
The aim of this project is to train a neural net to predict such ‘flow lines’ onto random meshes created from the captured point clouds.
Dataset
The next step involved the creation of a homogeneous dataset from our 3D-scans. To make the data less heavy, we decided to flatten the 3D-scanned facade features into greyscale heightmaps which allow us to recreate the 3D-scanned meshes by using the greyscale images as so called ‘bump maps’, more commonly known from rendering software and the displacement of surfaces used for realistic materials.
Figure 9: Heightmaps – Flattened 3D-Scans | Source: Sensing Sublime, IAAC
The image above shows some example flattened meshes from Antoni Gaudi’s Casa Batlló and the ideal direction for a robotic toolpath flow drawn on top by hand.
We decided to use transfer learning and backpropagation to train a neural network. To test this idea, we would need to prepare a dataset based on pairs of greyscale heightmaps and matching line drawings.
Figure 10: Dataset – 500 generated Heightmaps | Source: Authors
With a simple Grasshopper script, we generated a set of 500 pairs of images.
Figure 11: Dataset – 500 generated line drawings | Source: Authors
Training
The first iteration was trained within Grasshopper with the use of the Owl plugin and backpropagation. After approximately 10 hours of training, the prediction of the neural network was showing a kind of direction but was very far away from a good result.
Figure 12: Validation – 10 hours of training in Grasshopper | Source: Authors
The image above shows the validation of the first iteration. The first of two of the three highlighted images are the pre-generated pairs. The first is the heightmap and is our validation input for the neural network. The central image shows the so-called ‘ground truth’ and is basically what we would expect the network to output. The last image what the trained neural net actually predicts, and in the above image the ground truth and the prediction are very far away from each other.
Figure 13: Validation – 240 hours of training in Grasshopper | Source: Authors
After 10 days of training in the same Grasshopper environment the neural net made some significant progress. But the results are still not clear enough to be used as input for a robot toolpath.
Figure 14: Validation – 240 hours of training with Tensorflow GPU | Source: Authors
The next idea was to step up the training capacity by using Tensorflow for faster GPU processing. The decision was to train the network with the pix2pix transfer learning within Google Colab.
Figure 15: Transfer Learning – Training within Google Colab
The training within Colab achieved already very promising results after a few hours of training.
Figure 16: Validation – Comparison of training progress | Source: Authors
Eventually, the training went on for around one week and the predictions for the line drawings were mostly correct or close enough to predict a flow pattern for further toolpath creation.
Figure 17: Toolpath – From Mesh and Line drawing to robot toolpath | Source: Authors
The developed process can now be used to input any 3D-scan as flattened heightmap and be able to predict a line drawing for it. The line drawing acts similar to isocurves on NURBS surfaces and is in their flow-line-aesthetic used as robot toolpath for the sand casting process by tween between those lines and final be projected onto the mesh. This allows us to use any scanned architectural feature to be processed very fast for the robotic fabrication as a sand cast concrete panel.
Application
The potential application as an affordable and fast alternative for the fabrication of unique facade panels will be further investigated within the third term Studio project Sandcast.
Figure 18: Application – Complex facade panels | Source: Authors
Artificial Toolpath // S.3 Software is a project of IAAC, Institute for Advanced Architecture of Catalonia developed at the Master in Robotics and Advanced Construction in 2021 by:
Students: Hendrik Benz, Alberto Browne, Michael DiCarlo, Orestis Pavlidis
Faculty: Mateusz Zwierzycki
Faculty Assistant: Soroush Garivani
