Support for 16-Bit Floats

[GiorgosXou]

So recently, I started playing around with tensorflow again, and realised there's tf.keras.backend.set_floatx('float16'). So why not?

• https://forum.arduino.cc/t/16-bit-floats/315961/9
• https://github.com/RobTillaart/float16

[GiorgosXou]

keras.mixed_precision.set_dtype_policy('mixed_float16')

[GiorgosXou]

This feel so wrong even though it's working... ah... so, by tweeking this lovely float16 library, adding those:

  operator float() const;
  bool     operator < (const int& i);
  float16  operator / (const int& i);
  float16  operator / (const long int& li);
  float16  operator / (const unsigned int& ui);
  float16  operator * (const double& d);

^{(I imagine probably in the worst way I could implement them)}
And by making some tiny tweeks to the library such as:

#define DFLOAT float16
#define DFLOAT_LEN 4
// (and some castings)

I was able to run a NN with float16s (but not train one which is fine anyways)

[GiorgosXou]

the sketch uses 2k more program space though.

[GiorgosXou]

actually 1796-bytes

[GiorgosXou]

Feels so illegal but it's practical anyways... so, by removing NAN & INF cases from f16tof32 and f32tof16 you can save 86 bytes
^{(Considering you won't train a NN directly on the MCU [meaning potential "Exploding Gradients" etc.] )}

[GiorgosXou]

So, I used chatgpt to improve the implementation and it came up with this:

float float16::f16tof32(uint16_t _value) const
{
  uint16_t sgn = (_value & 0x8000);
  int exp = (_value & 0x7C00) >> 10;
  uint16_t man = (_value & 0x03FF);

  if (_value == 0) {
    return sgn ? -0.0f : 0.0f;
  }

  if (exp > 0) {
    // if (exp == 31) {
    //   return man ? NAN : (sgn ? -INFINITY : INFINITY);
    // }
    return ldexpf(static_cast<float>(man) / 1024.0f + 1.0f, exp - 15) * (sgn ? -1.0f : 1.0f);
  }

  return ldexpf(static_cast<float>(man) / 1024.0f, -24) * (sgn ? -1.0f : 1.0f);
}

I removed nan and inf again and managed to use just 1230-bytes extra.

[GiorgosXou]

Chat-GPT helped again:

float float16::f16tof32(uint16_t _value) const {
  uint16_t sgn = _value & 0x8000;
  int exp = (_value & 0x7C00) >> 10;
  uint16_t man = _value & 0x03FF;

  if (_value == 0) {
    return sgn ? -0.0f : 0.0f;
  }

  float result;
  if (exp > 0) {
    // if (exp == 31) {
    //   return man ? NAN : (sgn ? -INFINITY : INFINITY);
    // }
    result = ldexpf(static_cast<float>(man) / 1024.0f + 1.0f, exp - 15);
  } else {
    result = ldexpf(static_cast<float>(man) / 1024.0f, -24);
  }

  return sgn ? -result : result;
}

down to 1210-bytes extra

[GiorgosXou]

int-quantization solves this in a better way IMHO.... so i'm not 100% sure if I should continue with this...