io - nate - nca paper

A conversation with @[email protected] about:

Neural Cellular Automata Manifold

Alejandro Hernandez, Armand Vilalta, Francesc Moreno-Noguer

1. #Overview (initial analysis of the paper)
2. #Nate
3. #Themanual4am

---

overview

objective

The purpose of this exercise is to challenge the {approach; methods; phenomena-invariant model} central to this project, by application to this technical NCA paper (a domain which I am not familiar)

approach

The approach might be described reckon, reason, reconcile:

• First pass is {intuition; broad and-or general contextual alignment}
• Then built-up or built-in, iteratively, while ensuring compliance with initial constraints

like establishing working integration tests firsts, before embarking on further development/ refactor proper

status

I haven’t finished working through the technical implementation, as I have a few general observations to check/ synchronise, and questions on assumptions.

---

observations

high-level

High-level model differences:

1. This project models dynamic causal phenomena in 4D
   1. Relation between constraint, structure and behaviour
      1. All behaviour is relative to structure
      2. Constraint and structure/ behaviour are somewhat synonymous, translatable
         1. Possibility-space to specific form/ mutation
2. NCA models static images in 2D
   1. Constraint, structure
      1. Objective is to replicate image
      2. Though images are ‘revealed more than grown’
         1. Cells don’t develop an organism like this IRL
         2. Somewhat backwards, any intermediate, however legal?
      3. Colour implicates structure (colour is adornment to my model. Colour will be addressed below)

Questions:

1. is the objective of this line of experiments (NCA, etc) to eventually scale-up to real phenomena? 1. musing, seems that’s where i’m coming from?

latent and embedding spaces

• in zonal ban i touch on conceptual-spaces deriving from our internal maps of physical-environments
• in de-duplication i describe a simple mechanistic explanation of several forms of cognitive intelligence, as a result of present and sufficient representational symbolic manipulation

On reading this paper, I generally perceive:

1. Latent space as an approximation of the relational dimension of any conceptual-space
   1. Flattened to relative result/ set
2. Embedding space as the result of de-duplication
   1. Deep/ graph

(to be explored further below)

Questions:

1. does is this seem right?

Further, I perceive:

1. Distinct dimensions of both latent and embedding space
   1. Structural
   2. Behavioural
   3. Adornment (eg. colour)
2. Structure, behaviour
   1. Each structure has a unique behavioural latent space
      1. (Albeit with shared behavioural embedded space)
      2. Such that behavioural embedding might apply to plural distinct structures/ latent spaces
      3. Embeddings are like structural or behavioural metaphors
   2. Any embedding also exists within latent space
      1. Any node in a graph is a reference-frame for a set

Questions:

—what is the relation between latent space and embedding space for NCA?

legality

not all structures (or behaviours) are legal

1. IRL not all structures are legal
   1. Constraints
      1. Possibility-space
   2. Circumstantially evaluated structure/ structural-relation, informs legality
2. So IRL, constraints are conditional (to extrinsic circumstances), unique to each form, not homogeneous

Consider that where an image is wrong.

1. The delta exists with space-of-possible permissible variation/ different structures
2. The way in which an image is allowed to be wrong is instructive on space-of-possible variation/ different structures
   1. Permitted by the model in this case
   2. IRL, structures are validated/ invalidated intrinsically & extrinsically
      1. Actually, as a result of legality of respective space-of-possible/ permissible behaviour, of prior structure

purpose of NCA

—is legality the objective target of the NCA process? —to define constraints/ possibility space at each step? —is this constraint rule set homogeneous (flattened/ set-like), or nested/ branched (graph-like)? —when you ‘generate an image, even one it was not trained on’, do you mean: given a new image retrospectively, iteratively interpolate from single pixel, using pre-existing pixel transformations?

1. Generally, this seems like trying to determine which
   1. {Cell position you are in an organism; constituent molecule you are in a cell} by immediate peers
      1. NCA is like growing a cell rather than an organism?
         1. I might be influenced by the first paper with the lizard
            1. Cell growth to a complete form
            2. Rather than organism growth in scale/ form (which is complete from the beginning)
      2. —is this right?
   2. Whereby peers are relative environment
2. However IRL, organism growth changes each cells relative exposure to environment
   1. The space-of-possible at each step ought to change
   2. Otherwise the priors for each subsequent step are ‘sequentially out of bounds’ (?)
   3. IRL the legal space-of-possible at each step is finite, not ‘any intermediate form between’

notes

1. What is the fundamental learning-point of this intentionally simplified approach?
   1. Establish a common set of rules which allow for iterative revealing of a provided image?
      1. Which complies with constraints
      2. Not really to generate something new?

---

nate

1

https://tech.lgbt/@ngaylinn/111584537756510594

@Themanual4am It’s very interesting to see your analysis, as always. :).

You understand NCA models very well, but let me address some of your uncertainties to make things clearer.

First off, both of these papers are models of morphogenesis where many cells each contribute to a whole by using only locally available information to change their own state (color). I think the purpose of this is mostly to explore the potential of computational problem solving using many locally cooperating agents rather than a global top-down system. It also explores robustness. Each cell can handle a range of noisy or atypical conditions and tries to reestablish a “normal” baseline.

NCAs are an early exploration of such a system. They still feel a bit like a toy. Generating emojis isn’t useful, but there are some practical applications for creative work, such as generating patterns, textures, and spatial structures for art, video games, or movies. There’s no killer use case beyond that yet, but who knows what’s next.

---

2

https://tech.lgbt/@ngaylinn/111584538483733968

@Themanual4am This is not a perfect model of morphogenesis. The resemblance lies in the idea that each cell has the same “instructions,” and it follows those instructions based on locally available context / signals to contribute part of a coherent whole, without a top-down “blueprint” to follow. But the virtual cells are not much like real cells, the virtual body isn’t much like a real body, and the process of evolution is replaced with gradient descent. There is no movement, there is no behavior beyond form, and there is no “external environment” except for the loss function.

Still, the image is the emergent product of many cells “behaving” autonomously according to a learned program. When the image produced by this system is “wrong,” the training process modifies the genetic “language” used to describe a target image and / or the way that language gets interpreted into cell behavior, such that the system as a whole is more likely to draw the right image in the future.

---

3

https://tech.lgbt/@ngaylinn/111584539165044624

@Themanual4am In the Manifold paper, the genetic language represents the latent space for generating an image. The language implicitly defines a space of possible behaviors for each cell, and thus a space of possible images those cells can produce collectively. If the system is trained just on smiley emojis, then it would not be able to generate a spider web emoji because that would be illegal / inexpressible in the genetic language.

Embedded in this manifold are behavioral programs that capture different features of the trained images. This is a form of deduplication, where generally useful “building block” behaviors (like drawing an eye) are reused as part of more complex behaviors (like drawing a face). They are given a symbolic “name” in the genetic language, so they are easy to find and reuse when the system is faced with a new target image.

---

4

https://tech.lgbt/@ngaylinn/111584542945925232

@Themanual4am What happens when an NCA sees a novel image? Here we see a big difference from the original paper. In that one, the system must be trained how to grow the target image starting with a single pixel. In this new system, you could do the same thing, and it would be much faster: the genetic language provides an optimized search space. As long as the new emoji is expressible in that language, it should be quick to find.

However, the authors of the new paper took it even one step farther. They added an extra neural network that predicts what the genetic encoding of any image would be. That’s not strictly necessary, but it makes it so you can take a novel target image and immediately produce a genetic representation that approximately produces the new image without additional training.

I think if there’s any further work to be done here, it probably lies in exploring how this design change affects learnability and evolutionary dynamics. Perhaps I can think of a good way to do that.

---

themanual4am

https://mastodon.online/@themanual4am/111585972120465073

This is a fantastic breakdown Nate, thanks.

aside on circumstances ^{1 2 3}

I feel like there’s something I can’t quite articulate yet, which relates to the difference between ‘innate complexity’ and ‘contrived complexity’ ⁴, but a number of the questions and thoughts below touch on it:

• #Embeddings
• #Nca
• #General approach

embeddings

1. Their nature
   1. Within the embedding must exist sufficient information for each pixel to determine its own state based upon immediate peers
   2. —where is the conditionality for execution branching? —embedding evaluation?
      1. —extrinsic to embedding? —intrinsic? —both?
   3. If we assume some part is a heuristic to match adjacent peers with a generator which then specifies pixel value
      1. —is this scoped?
      2. —might any embedding apply at any moment?
         1. —does something have to iterate the whole space each time?
         2. —bail on first match? —some reductive weighting?
      3. If none match: —is there some kind of fallback?
2. —what is seen of this embedding space?
   1. —what size?
      1. —is it substantially smaller than an image?
3. —can we get at these embeddings?
   1. —are contents visible/ internal/ navigable?

• I think that real embedding are

  • Not open-problem-space, but (closed) puzzle space
    • Informed by substrate constraints
    • And scoped
  • A kidney cell has its scope reduced to kidneys, no longer needs (has available) other capability
    • Generally? circumstantially?

• Consider smiley facial features as organs

  • Eye pixels can only be eyes, not something else by mistake (whereas CA is ‘whatevs’?)
  • So one implicit observation of organism gene expression/ organ cells, is that ‘across an organism’, cellular scopes must be sufficiently reconciled to ensure that
    • —dna expresses for the right scope?
    • New adaptations don’t ‘collide’
  • organs of the same organism must be reconciled/ constrained by organismic set-of-all, such that adaptations don’t result in (structurally similar but not identical) circumstantially evaluated gene expressions/ chemical sequences colliding #rewrite

• IRL, embeddings demand resources {construction; evaluation}

  • Evaluation of arbitrarily sized embedding implicates time/ latency
  • Scopes/ async are natures way of mitigating this kind of cognitive latency

nca

• On training feedback

• When image is wrong, is the feedback/ modification process:

  • —autonomous?
    • —metric?
  • —manual?

• Dna is circumstantially evaluated sequence

  • —is a sequence a manifold?
  • Perhaps sequential operation produces artefacts in situ, more like an unordered space
  • Importantly, cgat is substrate
    • Finite legal forms at every step
    • Not any combination thereafter
  • And knowledge of substrate constraint doesn’t have to be learned per se, it guides at evaluation time (chemistry)
    • Circumstances are shaped (artefacts in situ), chemistry happens

• Manifold paper

  • Genetic language represents the latent space
    • —or the primitives of?
  • More is different ⁵

• The new paper ’extra step’

  • —like a best guess based upon pre-existing capability?

general approach

nate: “I think the purpose of this is mostly to explore the potential of computational problem solving using many locally cooperating agents rather than a global top-down system”

I think that not all problems can be solved in this way.

• Classic example of when extrapolating from a simplification, the question ought always be: —is it the right one?

• I think that problems need to be set-up to be solved in this way

  • Need to define the problem specifically
  • Like simplifying a tower structure to a one dimensional line, failing to recognise that the tower only manifests into a stable form by specific repeated intersection across dimensions
    • Any attempt to explore what might be able to be placed on the tower is missing clues which would simplify the task

• Can’t get to anywhere from anywhere

  • The wrong abstraction is contrived complexity
  • something which better matches reality may initially increase innate complexity, but innate complexity are clues. reducing contrived complexity, of the wrong abstractions, clears the way, to see more easily (and increases opportunity for subsequent application of ideas, because the process better aligns with reality) #rewrite