[Research] Audio support

[ChinaLiuKang]

I want to develop audio based on the project, where should I start first?

[drauggres]

Hi.

Android (sndcpy)

1. Examine https://github.com/rom1v/sndcpy (or maybe there are other similar project)
2. Find out what audio format it produces: compare it with https://en.wikipedia.org/wiki/HTML5_audio, to confirm that it is supported by your targer browser (also this could be helpful)
3. If you want to send audio data over Web Socket (as video data sent here), you may want to integrate WebSocket server into sndcpy jar (as I did with scrcpy: https://github.com/NetrisTV/scrcpy/tree/feature/websocket-server) or write a standalone WS server yourself (for example on node.js)
4. Read about Media Source Extensions
5. Look at https://github.com/samirkumardas/jmuxer, for example
6. https://github.com/GoogleChromeLabs/web-audio-samples

iOS (quicktime video hack)

This project for streaming uses ws-qvh. It uses QVH library. Audio stream produced by qvh is ignored. See:

• https://github.com/NetrisTV/ws-qvh/blob/a9a487ad1da187b0fa40b7888907f3e93d34fc35/naluwriter.go#L37
• https://github.com/danielpaulus/quicktime_video_hack/blob/7067c1d42ac6fa69c6e8ded5d3eb993affb895eb/screencapture/coremedia/avfilewriter.go#L86

See above for possible implementations of the client.

[ChinaLiuKang]

These suggestions have helped me a lot, thank you very much!!!

[denizkoele]

@ChinaLiuKang @lly835 did you managed to get this work, and might share the info? :)

[ChinaLiuKang]

Are you interested in perfecting the audio?

[drauggres]

You are welcome.
No, I don't really need audio for my use case, but if you manage to make it work - let me know and feel free to send a PR.

[yl12053]

I did find a way to do that, but because I am not familiear with Node.JS's web server & web socket, I might could not finish it by myself, so I will show what i have done so far here, shall I?
by examine the sndcpy script,
L27: %VLC% -Idummy --demux rawaud --network-caching=0 --play-and-exit tcp://localhost:%SNDCPY_PORT%
Here, the format is determined: PCM (16bit integer, Raw Audio)

and i have write a simple python script for that

import asyncio, websockets, base64, socket, random, os, queue, threading, time

devices_on = dict()
loop = asyncio.get_event_loop()

def device_thread(funcs, i, p):
    os.system("adb -s %s appoops set com.rom1v.sndcpy PROJECT_MEDIA allow" % i)
    os.system("adb -s %s forward tcp:%d localabstract:sndcpy" % (i, p))
    os.system("adb -s %s shell am start com.rom1v.sndcpy/.MainActivity" % i)
    time.sleep(2)
    s = socket.socket(2, 1)
    s.connect(("localhost", p))
    flag = False
    while True:
        v = s.recv(96000)
        if not flag:
            if v == b'':
                if flag:
                    continue
            else:
                flag = True
                #print("Frame got")
        #print("Sending %d bytes of data to %d functions"%(len(v), len(funcs)))
        for n in funcs:
            n(v)
        if v == b'':
            if last_zero < 1:
                last_zero = time.time()
            if time.time() - last_zero > 10:
                for x in devices_on[i]:
                    x(False)
                del devices_on[i]
                os.system("adb -s %s forward --remove tcp:%d"%(i, p))
                break
        else:
            last_zero = -1

async def main(websocket, path):
    path = path[1:]
    if path != "":
        dev = base64.b64decode(path.encode()).decode()
        que = queue.Queue()
        def app(data):
            que.put(data)
        if dev not in devices_on:
            port = random.randint(50000, 65535)
            while True:
                try:
                    s = socket.socket(2, 1)
                    s.bind(("localhost", port))
                    s.close()
                    break
                except:
                    port = random.randint(50000, 65535)
            devices_on[dev] = []
            threading.Thread(target=device_thread, args=(devices_on[dev], dev, port)).start()
            async def wait():
                while dev not in devices_on:
                    await asyncio.sleep(0.01)
            await wait()
        devices_on[dev].append(app)
        while True:
            itm = await asyncio.to_thread(que.get)
            if itm == False:
                await websocket.close()
                break
            else:
                if itm != b'':
                    await websocket.send(bytearray(itm))
                else:
                    await asyncio.sleep(0.01)
    else:
        await websocket.send("error: you must select a specific path first")
        await websocket.close()

loop.run_until_complete(websockets.serve(main, 'localhost', 8085))
loop.run_forever()

Basic idea is to determine the corresponding device by the path of the ws (in this case, the base64 encoding of the device serial),
and then do the sndcpy procedure:

1. Preserve a port in localhost for sndcpy communication
2. Forward the port to android device
3. Start main activity
   The python script allows multiple connection by creating a thread for each device (This is a bad approach), and then sending the data received to the callback which enqueue the data into the mainloop

For the client, i have used a library
pjky/pcm-player with the following code modified from their sample:

var player = null;
function loadPCM() {
      var socketURL = 'ws://localhost:8085/MTkyLjE2OC4zLjgyOjU1NTU=';
      //MTkyLjE2OC4zLjgyOjU1NTU=: 192.168.3.82:5555
      player = new PCMPlayer({
        inputCodec: 'Int16',
        channels: 2,
        sampleRate: 48000,
        flushTime: 0,
        onstatechange:(node, event, type)=>{
          console.log('type', node, event, type)
        },
        onended:(node, event)=>{
          console.log('end fire', node, event)
        }
      });

      var ws = new WebSocket(socketURL);
      ws.binaryType = 'arraybuffer';
      ws.addEventListener('message', function (event) {
        player.feed(event.data);
      });
    }
loadPCM();

(Yeah, i know, the indentation is bad)
Note that if using this approach, "pause" should not be used, instead volume should be set to 0
With this 2 tools, I successfully played audio by using websocket + sndcpy, but
it takes time to convert it into Node.JS

By the way, now, scrcpy2 support audio but only for Android 11+

[ChinaLiuKang]

ok！！！

[Raghavendra617]

Hello @ChinaLiuKang
Any luck on streaming Audio on iPhone during Mirroring

[Raghavendra617]

@drauggres suggest me a procedure to stream Music Audio out of iPhone during screen mirroring ?

[drauggres]

@Raghavendra617 Everything is in my first comment in the "iOS" section.