first commit

This commit is contained in:
Fabio 2026-10-03 12:16:58 +02:00
commit ad13739883
555 changed files with 655627 additions and 0 deletions

1
.Fabio.last_event Normal file
View file

@ -0,0 +1 @@
1774133507738

44
.env Normal file
View file

@ -0,0 +1,44 @@
BASE_URL=https://prova.patachina.it
SERVER_PORT=4000
# cartella dove scrivere gli output HLS (relativa alla root del progetto)
HLS_BASE=./hls_output
EMAIL=fabio@gmail.com
PASSWORD=master66
JWT_SECRET=123456789
JWT_EXPIRES=1h
# Dove si trova la cartella public (relativa alla root del progetto)
WEB_ROOT=public
# Percorso relativo di index.json dentro public/
INDEX_PATH=photos/index.json
# true = restituisce path assoluti nei record
PATH_FULL=true
# Logging
LOG_MODE=both # console | file | both
LOG_FILE=scan.log
LOG_DIR=public
LOG_VERBOSE=true
# POLLING_TIME in secondi
# Funzionamento:
# - POLLING_TIME = 0 → polling disattivato (solo WebSocket)
# - POLLING_TIME > 0 → esegue incrementalSync() ogni N secondi
# Perché esiste?
# - Serve come "rete di sicurezza" se il WebSocket cade,
# se la tab viene sospesa, o se qualche evento viene perso.
# - Google Photos, Dropbox e iCloud usano lo stesso approccio.
GALLERY_REFRESH_SECONDS=30
WS_PORT=4002
WS_HOST=0.0.0.0
PHOTO_RETENTION_DAYS=30
MAPBOX_PUBLIC_TOKEN=pk.eyJ1IjoicGlwcG9jYWwiLCJhIjoiY2xka28zdWVjMHBkZzNwbnoyMmlxYjA3MyJ9.9D5EB0SWydvAQZr-SdI64Q

7
.gitignore vendored Normal file
View file

@ -0,0 +1,7 @@
node_modules/
thumbs/
db.json
face-scan-daemon.lock
.venv-clustering/

BIN
IMG_0124.JPG Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 MiB

BIN
IMG_0125.JPG Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

BIN
IMG_20210602_100637.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.8 MiB

BIN
PIPPO.JPG Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 3 MiB

129
README.md Normal file
View file

@ -0,0 +1,129 @@
# Galleria con json-server e protetto con JWT S6J
## Installazione
clonare questa repo e installare tutte le dipendenze con `npm ci`
## Start/Stop servers
| Description | Script |
| ------------------------- | -------------------- |
| Start server senza auth | `npm start-no-auth` |
| Start server con auth | `npm run start` |
## Tools
| Description | Script |
| ------------------------------ | ------------------- |
| Generate user hashed passwords | `npm run hash` |
[json-server api reference](https://github.com/typicode/json-server)
## Come usarlo
clonare e poi installare con
```
npm ci
```
nel file .env ci sono tutti i dati da modificare
poi inserire in user.json user e password utilizzati per fare il login
la password da inserire è criptata e viene generata con npm run hash
il nome viene utilizzato come cartella da scansionare, si trova dentro photos
es:
```
name: Fabio
public/photos
└── Fabio
└── original
└── 2017Irlanda19-29ago
├── IMG_0092.JPG
├── IMG_0099.JPG
├── IMG_0100.JPG
```
poi dentro Fabio genererà thumbs con tutti i thumbs
- npm run start
- su IP:4000 ci sarà la galleria e andando su impostazioni si potrà fare lo scan di tutte le foto
dopo aver fatto lo scan è possibile richiedere il json al server con tutte le informazioni anche senza autorizzazione
basta farlo partire con npm run start-no-auth e le info si possono vedere con
ip:4000/photos
- npm start
---
#### FIT parse (utilizzo di python)
Per avere il parse dei fit installare python, pipx e i seguenyi packages
```
sudo apt install pipx
pipx ensurepath
pipx install fitdecode
```
usa per sapere dove è installato python
```
pipx list
```
in api_v1/scanner/parsers/parseFit.js va inserito il riferimento
in questo caso
```
async function parseFit(absPath) {
const frames = await new Promise((resolve, reject) => {
execFile(
"/home/orangepi/.local/pipx/venvs/fitdecode/bin/python",
["api_v1/scanner/parsers/parse_fit.py", absPath],
{ maxBuffer: 1024 * 1024 * 50 }, // 50 MB
```
Inspired in this [post](https://www.techiediaries.com/fake-api-jwt-json-server/) by [Techiediaries](https://www.techiediaries.com/)
#### Interrogazioni manuali
per vedere il token
```
curl -s -X POST https://prova.patachina.it/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"fabio@gmail.com","password":"master66"}'
```
per salvare il token
```
TOKEN=$(curl -s -X POST https://prova.patachina.it/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"fabio@gmail.com","password":"master66"}' \
| jq -r '.token')
```
per interrogare le foto
```
curl -s "https://prova.patachina.it/photos/" \
-H "Authorization: Bearer $TOKEN"
```
per interrogare le attività
```
curl -s "https://prova.patachina.it/activities/" \
-H "Authorization: Bearer $TOKEN"
```

278
README_WS.md Normal file
View file

@ -0,0 +1,278 @@
Perfetto Fabio — ora ti preparo il diagramma del flusso completo, chiaro, leggibile e fedele al comportamento reale del tuo sistema.
È un diagramma architetturale, non un semplice schema: mostra tutto il ciclo di vita di una sessione WebSocket affidabile, con SQLite, retry, timeout, needfullsync e sync del client.
---
🟦 DIAGRAMMA DEL FLUSSO COMPLETO — WebSocket Affidabile + Sync
Lo divido in 6 sezioni:
1. Connessione iniziale
2. Autenticazione + session_id
3. Invio evento dal server
4. Ricezione evento dal client
5. Retry + Timeout + needfullsync
6. Riconnessione del client
---
🟩 1) CONNESSIONE INIZIALE
`
CLIENT (browser) SERVER (ws-server.js)
──────────────────────────────────────────────────────────────────────
startWebSocket()
↓
Apre WebSocket ------------------------→ ws.on("connection")
crea ws object
ws.authenticated = false
`
---
🟩 2) AUTENTICAZIONE + SESSION_ID
`
CLIENT SERVER
──────────────────────────────────────────────────────────────────────
Invia:
{
type: "auth",
token: JWT,
session_id: <persistente>
}
↓ ws.on("message")
verifica JWT
cerca sessione in SQLite
se non esiste → createSession()
aggiorna last_ack
salva wsBySession[session_id] = ws
Riceve:
{
type: "auth_ok",
user: "...",
session_id: "...",
needfullsync: true/false
}
↓
Se needfullsync = true → incrementalSync()
`
---
🟩 3) INVIO EVENTO DAL SERVER (broadcast affidabile)
`
SERVER
──────────────────────────────────────────────────────────────────────
broadcastToUserReliable(user, payload)
↓
Genera event_id (UUID)
↓
Per ogni sessione dell’utente:
- Se ws attivo:
ws.send(payload + event_id)
addPendingEvent(eventid, sessionid)
- Se ws NON attivo:
setSessionNeedFullSync(session_id, true)
`
---
🟩 4) RICEZIONE EVENTO DAL CLIENT
`
CLIENT
──────────────────────────────────────────────────────────────────────
Riceve evento:
{
type: "added" | "removed" | ...,
id: "...",
event_id: "..."
}
↓
Se event_id già in processedEvents:
- invia solo ACK
- ignora evento
- FINE
Se nuovo:
- applica evento (add/remove foto)
- aggiorna UI
- aggiunge event_id a processedEvents
- invia ACK:
{ type: "ack", event_id }
`
---
🟩 5) RETRY + TIMEOUT + NEEDFULLSYNC
`
SERVER (ogni 3 secondi)
──────────────────────────────────────────────────────────────────────
Legge da SQLite tutti gli eventi pendenti
con sent_at < now - 3000 ms
Per ogni evento:
Se retries < 40 (2 minuti):
- se ws attivo:
ws.send(payload)
updatePendingEventRetry()
- se ws NON attivo:
setSessionNeedFullSync(session_id, true)
deletePendingEvent(event_id)
Se retries >= 40:
- ws.terminate()
- wsBySession.delete(session_id)
- deleteSession(session_id) ← cancella queue
- (sessione verrà ricreata al prossimo auth)
`
---
🟩 6) RICONNESSIONE DEL CLIENT
`
CLIENT riapre la tab / si risveglia
──────────────────────────────────────────────────────────────────────
startWebSocket()
↓
Invia auth con lo stesso session_id
↓
SERVER:
- trova sessione in SQLite?
NO → la ricrea con needfullsync = true
SÌ → legge needfullsync
- invia authok con needfull_sync
CLIENT:
Se needfullsync = true:
incrementalSync()
Altrimenti:
continua con WS realtime
`
---
🟦 RISULTATO DEL FLUSSO
✔ Nessun evento perso
✔ Nessun evento duplicato
✔ Chrome Android non può più “morire in silenzio”
✔ Multi‑dispositivo perfetto
✔ Queue pulita dopo 2 minuti
✔ Sync completo automatico quando serve
✔ WS leggero (ping solo on‑demand)
✔ Stato persistente in SQLite
---
┌──────────────────────────┐ ┌──────────────────────────┐
│ CLIENT │ │ SERVER │
└─────────────┬────────────┘ └─────────────┬────────────┘
│ │
│ 1) Connessione WS │
├──────────────────────────────────────→│
│ │
│ 2) AUTH + session_id │
│ {type:auth, token, session_id} │
├──────────────────────────────────────→│
│ │
│ │ verifica JWT
│ │ cerca sessione in DB
│ │ crea/aggiorna sessione
│ │
│ 3) AUTH_OK │
│ ←──────────────────────────────────────┤
│ {need_full_sync:true/false} │
│ │
│ se need_full_sync → incrementalSync() │
│ │
──────────────┼────────────────────────────────────────┼──────────────
│ │
│ 4) Ricezione evento WS │
│ {type:added/removed, event_id} │
│ ←──────────────────────────────────────┤
│ │
│ se event_id già visto → ACK │
│ se nuovo → applica evento │
│ │
│ ACK │
├──────────────────────────────────────→│
│ │
──────────────┼────────────────────────────────────────┼──────────────
│ │
│ │ 5) Retry loop (ogni 3s)
│ │ cerca eventi pendenti
│ │
│ │ se retries < 40:
│ │ reinvia evento
│ │
│ │ se retries >= 40:
│ │ ws.terminate()
│ │ deleteSession()
│ │ (queue eliminata)
│ │
──────────────┼────────────────────────────────────────┼──────────────
│ │
│ 6) Riconnessione client │
├──────────────────────────────────────→│
│ {auth, token, session_id} │
│ │
│ │ server vede sessione mancante
│ │ → need_full_sync = true
│ │
│ AUTH_OK │
│ ←──────────────────────────────────────┤
│ {need_full_sync:true} │
│ │
│ incrementalSync() │
│ │
──────────────┴────────────────────────────────────────┴──────────────
CLIENT → WS CONNECT
CLIENT → AUTH(token, session_id)
SERVER → AUTH_OK(need_full_sync)
EVENTO:
SERVER → EVENT(event_id)
CLIENT:
se nuovo → applica + ACK
se già visto → solo ACK
RETRY LOOP:
se retries < 40 → reinvia
se retries >= 40 → chiudi WS + cancella sessione
RICONNESSIONE:
CLIENT → AUTH(session_id)
SERVER → AUTH_OK(need_full_sync=true)
CLIENT → incrementalSync()
______
AUTH → EVENTI → ACK → (retry finché vivo)
se client sparisce → timeout → cancella sessione
se torna → need_full_sync → incrementalSync

761
RICONOSCIMENTO_SERVER.md Normal file
View file

@ -0,0 +1,761 @@
# Riconoscimento facciale nel server s141js
## Scopo del documento
Questo documento descrive il funzionamento attuale del riconoscimento facciale nel server `s141js`.
La soluzione usa due percorsi paralleli:
- Python produce il risultato ufficiale e lo salva nel database;
- OpenCL sulla GPU Mali-G610 ricalcola lo stesso risultato in modalità shadow e verifica la coerenza.
La modalità shadow non sostituisce ancora il risultato Python e non modifica le valutazioni ufficiali.
## Legenda dei segnaposto
Il file usa segnaposto al posto dei caratteri Markdown che possono essere alterati durante la copia in chat o nel terminale.
```text
# titolo livello 1, da convertire in #
## titolo livello 2, da convertire in ##
### titolo livello 3, da convertire in ###
#### titolo livello 4, da convertire in ####
__FENCE__ apertura o chiusura di un blocco di codice
__FENCE_TEXT__ apertura di un blocco text
__FENCE_BASH__ apertura di un blocco bash
__FENCE_JSON__ apertura di un blocco json
__FENCE_JS__ apertura di un blocco javascript
` apice rovescio singolo
```
## Architettura generale
```text
Fotografia
|
v
SCRFD tramite RKNN e NPU
|
v
Bounding box e cinque landmark
|
v
Allineamento del volto a 112 x 112
|
v
SFace tramite RKNN e NPU
|
v
Embedding float32[128] normalizzato L2
|
+-------------------------------+
| |
v v
Pipeline Python ufficiale kNN OpenCL shadow
| |
v v
Valutazioni salvate Confronto di coerenza
nel database Python contro OpenCL
| |
+---------------+---------------+
|
v
Report e stato del job
```
## Componenti principali
### Face engine RKNN
Il face engine usa la NPU del dispositivo RK3588 per:
- rilevare i volti con SCRFD;
- calcolare confidence, bounding box e landmark;
- allineare il volto;
- generare un embedding SFace;
- normalizzare l'embedding con norma L2.
File principali:
```text
include/face_config.h
include/face_engine.h
src/face_engine.cc
src/face_recognition.cc
src/face_scan.cc
```
Ogni embedding ha il seguente formato:
```text
Dimensione: 128 componenti
Tipo: float32
Dimensione binaria: 512 byte
Ordine byte: little-endian
Normalizzazione: L2
```
Poiché gli embedding sono normalizzati, la cosine similarity coincide con il prodotto scalare.
### Motore kNN nativo
Il motore nativo è installato in:
```text
native/face-knn/
```
File principali:
```text
native/face-knn/bin/face_knn
native/face-knn/kernels/knn_float4_topk.cl
native/face-knn/run-face-knn.sh
native/face-knn/config.json
native/face-knn/SHA256SUMS
```
Il launcher `run-face-knn.sh` imposta la directory di lavoro corretta prima di avviare il binario. Questo permette a `face_knn` di trovare il kernel tramite il percorso relativo:
```text
kernels/knn_float4_topk.cl
```
Il motore supporta due backend:
```text
cpu
opencl
```
Il backend CPU è il riferimento numerico e il fallback. Il backend OpenCL usa la GPU Mali-G610.
### Runtime Node.js
Il modulo:
```text
tools/faceKnnRuntime.js
```
ha queste responsabilità:
1. legge la configurazione da `native/face-knn/config.json`;
2. recupera gli embedding dal database SQLite;
3. preserva l'ordine delle query prodotto dall'esperimento Python;
4. rilegge i prototype attivi ordinati per `selection_order`;
5. verifica che ogni embedding sia un Buffer da 512 byte;
6. crea file binari temporanei per query e prototype;
7. genera la self-map per il leave-one-out;
8. avvia il binario nativo;
9. analizza il JSON prodotto;
10. elimina i file biometrici temporanei.
I file temporanei vengono creati sotto:
```text
var/face-knn/run-*
```
Durante l'esecuzione normale vengono rimossi automaticamente.
### Shadow runner
Il comando indipendente:
```text
tools/runFaceKnnShadow.js
```
confronta il risultato Python con quello del backend nativo.
Confronta per ogni query:
- prototype migliore;
- best similarity;
- media dei primi tre score;
- decisione recognized;
- esclusione leave-one-out.
Il report contiene solo metadati e risultati numerici. Non contiene gli embedding.
## Database e tabelle utilizzate
Il database attivo è:
```text
api_v1/database.sqlite
```
Le tabelle principali sono:
### faces
Contiene i volti e gli embedding da 512 byte.
### person_face_prototypes
Contiene i prototype attivi di ogni persona.
L'ordine è determinato da:
```text
selection_order
```
La corrispondenza con gli indici OpenCL è:
```text
selection_order 1 -> indice OpenCL 0
selection_order 2 -> indice OpenCL 1
selection_order 3 -> indice OpenCL 2
selection_order 4 -> indice OpenCL 3
selection_order 5 -> indice OpenCL 4
```
### face_prototype_jobs
Contiene i job di ricalcolo dei prototype.
Stati principali:
```text
pending
processing
done
error
superseded
```
### person_face_recognition_evaluations
Contiene le valutazioni ufficiali prodotte dalla pipeline Python.
Campi principali:
```text
face_id
prototype_job_id
prototype_count
best_similarity
top3_mean
best_prototype_face_id
recognition_threshold
recognized
evaluated_at
```
### face_recognition_index_state
Tiene traccia dello stato dell'indice persistente.
Stato attuale validato:
```text
user: Fabio
status: dirty
dirty: 1
version: 0
prototype_count: 5
index_path: null
metadata_path: null
built_at: null
last_error: null
```
Lo stato `dirty` indica che i prototype sono stati ricalcolati, ma l'indice persistente non è ancora stato costruito.
## Come nasce un job di ricalcolo
Il router è montato con il prefisso:
```text
/face-review
```
L'endpoint manuale è:
```text
POST /face-review/persons/:personId/prototypes/recalculate
```
La richiesta richiede:
```text
Authorization: Bearer TOKEN
Content-Type: application/json
```
Esempio di body:
```json
{
"reason": "opencl_shadow_validation"
}
```
La funzione `queuePrototypeRefresh()`:
- crea un job pending se non ne esiste uno;
- aggiorna il job pending esistente per evitare duplicati;
- salva reason, requested_by e timestamp.
Il ricalcolo viene accodato anche automaticamente quando:
- viene assegnato un cluster a una persona;
- un volto viene rimosso da una persona;
- un volto noise viene assegnato;
- un volto viene riassegnato;
- più volti noise vengono assegnati insieme.
## Elaborazione del job
Il worker è:
```text
tools/runPrototypeWorker.js
```
Esecuzione manuale:
```bash
node tools/runPrototypeWorker.js \
--persist \
--job-id JOB_ID
```
Il flusso è il seguente.
### 1. Acquisizione del job
Il worker cerca un job con stato `pending`, lo porta a `processing` e incrementa `attempts`.
### 2. Firma degli input
Il worker calcola una firma SHA-256 basata sui volti e sulle esclusioni di riferimento.
Se la firma cambia durante il lavoro, il job viene marcato `superseded` e viene creato un job sostitutivo.
### 3. Esperimento Python
Il worker avvia:
```text
tools/prototype_experiment.py
```
Parametri attuali:
```text
noise_prototypes: 2
greedy_additions: 2
min_detector_score: 0.60
```
L'esperimento produce:
```text
prototype_sets.greedy_gain
evaluations.greedy_gain.queries
evaluations.greedy_gain.summary
```
Ogni query Python contiene:
```text
face_id
photo_name
is_prototype
best_similarity
best_prototype_face_id
top3_mean
```
### 4. Selezione dei prototype
Il worker:
1. applica le esclusioni manuali;
2. mantiene i prototype bloccati;
3. disattiva i prototype automatici precedenti;
4. salva i nuovi prototype;
5. assegna `selection_order` a partire da 1.
Per Jessica l'ordine validato è:
```text
0 IMG_0145.JPG cluster_medoid
1 IMG_0141.JPG cluster_medoid
2 IMG_0193.JPG cluster_medoid
3 IMG_0143.JPG greedy_leave_one_out_gain
4 IMG_0146.JPG greedy_leave_one_out_gain
```
Il numero a sinistra è l'indice kNN, cioè `selection_order - 1`.
### 5. Valutazioni Python ufficiali
La soglia attuale è:
```text
0.50
```
Le valutazioni precedenti vengono sostituite atomicamente nella transazione SQLite.
Il risultato Python viene salvato in `person_face_recognition_evaluations` ed è ancora il risultato ufficiale.
### 6. Aggiornamento dello stato indice
Dopo il ricalcolo dei prototype:
```text
status = dirty
dirty = 1
prototype_count = numero dei prototype attivi
last_error = null
```
### 7. Chiusura della transazione
`persistResult()` conclude la transazione prima dell'avvio OpenCL.
Questo evita di mantenere un lock SQLite durante:
- creazione dei file binari;
- inizializzazione OpenCL;
- compilazione del kernel;
- esecuzione del kNN;
- scrittura del report shadow.
### 8. Shadow OpenCL
Dopo il commit della transazione, il worker chiama `runFaceKnnShadow()`.
La sequenza è:
```text
Risultato Python già salvato
|
v
Recupero embedding query nello stesso ordine Python
|
v
Recupero prototype ordinati per selection_order
|
v
Generazione queries.bin
Generazione prototypes.bin
Generazione self-map.txt
|
v
face_knn --backend opencl
|
v
Confronto Python contro OpenCL
|
v
Report shadow JSON
```
## Leave-one-out
Se una query è anche un prototype, il confronto con sé stessa deve essere escluso.
Il runtime costruisce una mappa tra `face_id` e indice del prototype.
La self-map contiene:
```text
-1 -> nessuna esclusione
0 -> escludi prototype indice 0
1 -> escludi prototype indice 1
2 -> escludi prototype indice 2
...
```
Il kernel calcola il TOPK completo. L'esclusione viene applicata sul risultato host.
## Output del motore nativo
Il binario mantiene separati i flussi:
```text
stdout -> JSON valido
stderr -> diagnostica OpenCL e messaggi del driver Mali
```
Il runtime Mali può scrivere direttamente:
```text
arm_release_ver: g24p0-00eac0, rk_so_ver: 10
```
La CLI devia temporaneamente stdout verso stderr durante OpenCL e lo ripristina prima di scrivere il JSON.
## Controlli shadow
Per ogni query vengono confrontati:
- prototype migliore Python e OpenCL;
- best similarity;
- media dei primi tre score;
- decisione recognized;
- leave-one-out.
Tolleranza configurata:
```text
0.00001
```
Campi aggregati del report:
```text
query_count
prototype_count
top1_matches
decision_matches
best_score_matches
top3_mean_matches
maximum_score_error
maximum_top3_mean_error
mismatch_count
```
I report automatici vengono scritti in:
```text
var/face-knn/shadow-JOB_ID.json
```
Il report non contiene embedding.
## Gestione degli errori
### Errore Python o persistenza
```text
Il job viene marcato error.
last_error viene valorizzato.
Il worker restituisce exit code non zero.
```
### Errore OpenCL shadow
```text
Il job Python resta done.
Le valutazioni ufficiali restano salvate.
L'errore viene registrato nel log shadow.
Non viene effettuato rollback.
```
### Mismatch Python/OpenCL
```text
Il job Python resta done.
Il report contiene mismatch_count maggiore di zero.
OpenCL non viene promosso a motore ufficiale.
```
## Configurazione attuale
File:
```text
native/face-knn/config.json
```
Configurazione validata:
```json
{
"enabled": true,
"mode": "shadow",
"backend": "opencl",
"fallbackBackend": "cpu",
"dimension": 128,
"topK": 5,
"recognitionThreshold": 0.5,
"scoreTolerance": 0.00001,
"autoBackendMinimumComparisons": 100000,
"launcher": "native/face-knn/run-face-knn.sh",
"indexDirectory": "api_v1/face-recognition-index",
"runtimeDirectory": "var/face-knn"
}
```
## Validazione completata
Persona validata:
```text
User: Fabio
Persona: Jessica
Person ID: 77456100-d7b2-4c36-b3d6-09f8c1c9d7e7
Query: 21
Prototype: 5
```
Risultato funzionale:
```text
Riconosciuti: 20
Sotto soglia: 1
Copertura: 95.2%
Caso sotto soglia: IMG_0183.JPG
Score: 0.4664280414581299
Miglior prototype: IMG_0143.JPG
Indice prototype: 3
```
Risultato shadow automatico:
```text
Top-1 identici: 21 / 21
Decisioni identiche: 21 / 21
Best score entro tolleranza: 21 / 21
Top-3 mean entro tolleranza: 21 / 21
Mismatch: 0
Errore massimo best score: 1.7869088742017425e-7
Errore massimo top-3 mean: 1.3911677043676463e-7
```
Job validato:
```text
Job ID: 7741c033-dae9-4b6f-b00f-bc082f08b8a1
Stato: done
Tentativi: 1
Reason: opencl_shadow_validation
Last error: null
```
## Stato attuale della produzione
```text
Python:
- seleziona i prototype;
- calcola le valutazioni;
- salva il risultato ufficiale.
OpenCL:
- ricalcola il kNN dopo il commit;
- confronta il risultato;
- genera il report shadow;
- non modifica il risultato ufficiale.
```
## Prossimi passi
1. creare il builder dell'indice persistente;
2. generare atomicamente embeddings e metadati;
3. portare `face_recognition_index_state` da dirty a ready;
4. mantenere la modalità shadow per più job reali;
5. definire criteri di promozione OpenCL;
6. mantenere il backend CPU come fallback;
7. passare a OpenCL ufficiale solo dopo validazioni sufficienti.
## Comandi operativi utili
### Controllare i job pending
```bash
sqlite3 -header -column \
api_v1/database.sqlite \
"
SELECT
id,
user,
person_id,
status,
reason,
requested_at
FROM face_prototype_jobs
WHERE status = 'pending'
ORDER BY requested_at;
"
```
### Eseguire un job specifico
```bash
node tools/runPrototypeWorker.js \
--persist \
--job-id "JOB_ID"
```
### Validare un report shadow
```bash
python3 -m json.tool \
"var/face-knn/shadow-JOB_ID.json" \
> /dev/null \
&& echo "OK: report shadow valido"
```
### Controllare le valutazioni ufficiali
```bash
sqlite3 -header -column \
api_v1/database.sqlite \
"
SELECT
COUNT(*) AS evaluations,
SUM(CASE WHEN recognized = 1 THEN 1 ELSE 0 END)
AS recognized_faces,
SUM(CASE WHEN recognized = 0 THEN 1 ELSE 0 END)
AS below_threshold
FROM person_face_recognition_evaluations;
"
```
### Controllare lo stato dell'indice
```bash
sqlite3 -header -column \
api_v1/database.sqlite \
"
SELECT
user,
status,
dirty,
version,
prototype_count,
index_path,
metadata_path,
built_at,
last_error
FROM face_recognition_index_state
WHERE user = 'Fabio';
"
```
## Conversione in Markdown standard
Per generare un file Markdown normale:
```bash
sed \
-e 's/^__H4__[[:space:]]*/#### /' \
-e 's/^__H3__[[:space:]]*/### /' \
-e 's/^__H2__[[:space:]]*/## /' \
-e 's/^__H1__[[:space:]]*/# /' \
-e 's/^__FENCE_TEXT__$/```text/' \
-e 's/^__FENCE_BASH__$/```bash/' \
-e 's/^__FENCE_JSON__$/```json/' \
-e 's/^__FENCE_JS__$/```javascript/' \
-e 's/^__FENCE__$/```/' \
-e 's/`/`/g' \
RICONOSCIMENTO_SERVER.src.md \
> RICONOSCIMENTO_SERVER.md
```
Verifica che non siano rimasti segnaposto:
```bash
grep -nE \
'__H[1-4]__|__FENCE|`' \
RICONOSCIMENTO_SERVER.md \
|| echo "OK: segnaposto convertiti"
```

BIN
VID_20260221_095917.mp4 Normal file

Binary file not shown.

BIN
a.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

4
api_v1/admin_secret.json Normal file
View file

@ -0,0 +1,4 @@
{
"email": "admin@gmail.com",
"password": "master66"
}

35
api_v1/config.js Normal file
View file

@ -0,0 +1,35 @@
require('dotenv').config();
module.exports = {
BASE_URL: process.env.BASE_URL,
EMAIL: process.env.EMAIL,
PASSWORD: process.env.PASSWORD,
SEND_PHOTOS: (process.env.SEND_PHOTOS || 'true').toLowerCase() === 'true',
// Cartella public (root dei file statici)
WEB_ROOT: process.env.WEB_ROOT || 'public',
// PATH_FULL ora è un boolean, non un path
PATH_FULL: (process.env.PATH_FULL || 'false').toLowerCase() === 'true',
// Estensioni supportate per FOTO/VIDEO
SUPPORTED_EXTS: new Set([
'.jpg', '.jpeg', '.png', '.webp', '.heic', '.heif',
'.mp4', '.mov', '.m4v'
]),
// Estensioni supportate per ATTIVITÀ (FIT/GPX/TCX)
SUPPORTED_ACTIVITY_EXTS: new Set([
'.fit',
'.gpx',
'.tcx'
]),
// 🔥 NUOVO: intervallo refresh gallery (in secondi)
GALLERY_REFRESH_SECONDS: Number(process.env.GALLERY_REFRESH_SECONDS || 30),
// 🔥 PORTA SERVER
PORT: parseInt(process.env.SERVER_PORT || "4000", 10)
};

BIN
api_v1/database.sqlite Normal file

Binary file not shown.

128
api_v1/geo.js Normal file
View file

@ -0,0 +1,128 @@
// api_v1/geo.js
const axios = require("axios");
async function loc(lng, lat) {
const primary = await place(lng, lat);
if (
primary &&
primary.city &&
primary.country &&
primary.address
) {
return primary;
}
const fallback = await placePhoton(lng, lat);
if (!primary) return fallback;
if (!primary.city && fallback?.city) {
primary.city = fallback.city;
}
if (!primary.postcode && fallback?.postcode) {
primary.postcode = fallback.postcode;
}
if (!primary.address && fallback?.address) {
primary.address = fallback.address;
}
if (!primary.region && fallback?.region) {
primary.region = fallback.region;
}
if (!primary.country_code && fallback?.country_code) {
primary.country_code = fallback.country_code;
}
if (!primary.county && fallback?.county) {
primary.county = fallback.county;
}
if (!primary.county_code && fallback?.county_code) {
primary.county_code = fallback.county_code;
}
return primary;
}
function normCountryCode(v) {
if (!v) return undefined;
const s = String(v).trim().toUpperCase();
return s.length ? s : undefined;
}
async function place(lng, lat) {
const apiKey = "6dc7fb95a3b246cfa0f3bcef5ce9ed9a";
const url = `https://api.geoapify.com/v1/geocode/reverse?lat=${lat}&lon=${lng}&apiKey=${apiKey}`;
try {
const r = await axios.get(url);
if (r.status !== 200) return undefined;
if (!r.data.features || r.data.features.length === 0) return undefined;
const k = r.data.features[0].properties;
return {
continent: k?.timezone?.name?.split("/")?.[0] || undefined,
// Country / Paese
country: k?.country || undefined,
country_code: normCountryCode(k?.country_code),
// County / contea
county: k?.county || undefined,
county_code: k?.county_code || undefined,
// Regione / stato amministrativo
region: k?.state || undefined,
postcode: k?.postcode || undefined,
city: k?.city || k?.town || k?.village || undefined,
address: k?.address_line1 || undefined,
timezone: k?.timezone?.name || undefined,
time: k?.timezone?.offset_STD || undefined,
};
} catch (err) {
return undefined;
}
}
async function placePhoton(lng, lat) {
try {
const url = `https://photon.patachina.it/reverse?lon=${lng}&lat=${lat}`;
const r = await axios.get(url);
if (!r.data || !r.data.features || r.data.features.length === 0) {
return undefined;
}
const p = r.data.features[0].properties;
return {
continent: undefined,
// Country / Paese
country: p.country || undefined,
country_code: normCountryCode(p.countrycode || p.country_code),
// County / contea, se disponibile
county: p.county || undefined,
county_code: p.county_code || undefined,
region: p.state || undefined,
postcode: p.postcode || undefined,
city: p.city || p.town || p.village || undefined,
address: p.street ? `${p.street} ${p.housenumber || ""}`.trim() : undefined,
timezone: undefined,
time: undefined,
};
} catch (err) {
return undefined;
}
}
module.exports = loc;

117
api_v1/geo.js.ok Normal file
View file

@ -0,0 +1,117 @@
const axios = require("axios");
async function loc(lng, lat) {
const primary = await place(lng, lat); // Geoapify
const fallback = await placePhoton(lng, lat); // Photon
if (!primary) return fallback;
if (!primary.city && fallback?.city) {
primary.city = fallback.city;
}
if (!primary.postcode && fallback?.postcode) {
primary.postcode = fallback.postcode;
}
if (!primary.address && fallback?.address) {
primary.address = fallback.address;
}
if (!primary.region && fallback?.region) {
primary.region = fallback.region;
}
if (!primary.country_code && fallback?.country_code) {
primary.country_code = fallback.country_code;
}
if (!primary.county && fallback?.county) {
primary.county = fallback.county;
}
if (!primary.county_code && fallback?.county_code) {
primary.county_code = fallback.county_code;
}
return primary;
}
function normCountryCode(v) {
if (!v) return undefined;
const s = String(v).trim().toUpperCase();
return s.length ? s : undefined;
}
async function place(lng, lat) {
const apiKey = "6dc7fb95a3b246cfa0f3bcef5ce9ed9a";
const url = `https://api.geoapify.com/v1/geocode/reverse?lat=${lat}&lon=${lng}&apiKey=${apiKey}`;
try {
const r = await axios.get(url);
if (r.status !== 200) return undefined;
if (!r.data.features || r.data.features.length === 0) return undefined;
const k = r.data.features[0].properties;
return {
continent: k?.timezone?.name?.split("/")?.[0] || undefined,
// Country / Paese
country: k?.country || undefined,
country_code: normCountryCode(k?.country_code),
// County / contea
county: k?.county || undefined,
county_code: k?.county_code || undefined,
// Regione / stato amministrativo
region: k?.state || undefined,
postcode: k?.postcode || undefined,
city: k?.city || k?.town || k?.village || undefined,
address: k?.address_line1 || undefined,
timezone: k?.timezone?.name || undefined,
time: k?.timezone?.offset_STD || undefined,
};
} catch (err) {
return undefined;
}
}
async function placePhoton(lng, lat) {
try {
const url = `https://photon.patachina.it/reverse?lon=${lng}&lat=${lat}`;
const r = await axios.get(url);
if (!r.data || !r.data.features || r.data.features.length === 0) {
return undefined;
}
const p = r.data.features[0].properties;
return {
continent: undefined,
// Country / Paese
country: p.country || undefined,
country_code: normCountryCode(p.countrycode || p.country_code),
// County / contea, se disponibile
county: p.county || undefined,
county_code: p.county_code || undefined,
region: p.state || undefined,
postcode: p.postcode || undefined,
city: p.city || p.town || p.village || undefined,
address: p.street ? `${p.street} ${p.housenumber || ""}`.trim() : undefined,
timezone: undefined,
time: undefined,
};
} catch (err) {
return undefined;
}
}
module.exports = loc;

View file

@ -0,0 +1,64 @@
// api_v1/scanner/debugVideoDates.js
const path = require('path');
const { probeVideo } = require('./video');
async function debugVideo(absPath) {
console.log("🎥 File:", absPath);
const info = await probeVideo(absPath);
console.log("\n=== RAW probeVideo(info) ===\n");
console.dir(info, { depth: 6 });
// Estrazione campi data come nello scanner
const formatTags = info.format?.tags || {};
const stream0Tags = info.streams?.[0]?.tags || {};
const stream1Tags = info.streams?.[1]?.tags || {};
const creationFormat = formatTags.creation_time || null;
const creationStream0 = stream0Tags.creation_time || null;
const creationStream1 = stream1Tags.creation_time || null;
console.log("\n=== DATE CANDIDATE ===");
console.log("format.tags.creation_time :", creationFormat);
console.log("streams[0].tags.creation_time:", creationStream0);
console.log("streams[1].tags.creation_time:", creationStream1);
// Simulazione logica attuale dello scanner (ma esplicita)
let videoDate =
creationFormat ||
creationStream0 ||
creationStream1 ||
null;
console.log("\nScelta videoDate (prima di qualsiasi conversione):", videoDate);
if (videoDate) {
// Variante 1: SENZA conversione (quella che ti ho suggerito)
const takenAtRaw = videoDate;
// Variante 2: CON conversione (quella che ti sballa il giorno)
const takenAtIso = new Date(videoDate).toISOString();
console.log("\n=== CONFRONTO ===");
console.log("takenAtRaw (usato così com'è) :", takenAtRaw);
console.log("takenAtIso (new Date().toISOString):", takenAtIso);
} else {
console.log("\n⚠ Nessuna data trovata nei tag video.");
}
}
async function main() {
const arg = process.argv[2];
if (!arg) {
console.error("Uso: node debugVideoDates.js /percorso/al/video.mp4");
process.exit(1);
}
const absPath = path.resolve(arg);
await debugVideo(absPath);
}
main().catch(err => {
console.error("❌ Errore debugVideoDates:", err);
});

View file

@ -0,0 +1,73 @@
// api_v1/scanner/deleteFolder.js
const path = require('path');
const fsp = require('fs/promises');
const { WEB_ROOT } = require('../config');
module.exports = function createDeleteFolderFunctions(db, RETENTION_DAYS = 30) {
const createCleanupFunctions = require('./orphanCleanup');
const { deleteFromDB, deleteThumbsById } = createCleanupFunctions(db, RETENTION_DAYS);
/**
* Cancella una cartella per un utente:
* - soft delete immediato di tutte le foto
* - hard delete se retention scaduta
* - registra hard delete in deleted_hard
* - elimina thumbs solo per hard delete
*/
async function deleteFolderForUser(userName, folderName) {
// 1) Recupera tutti gli ID nel DB
const rows = await db('photos')
.where({ user: userName, cartella: folderName })
.select('id');
let softCount = 0;
let hardCount = 0;
// 2) Soft delete / Hard delete per ogni foto
for (const r of rows) {
const id = r.id;
const result = await deleteFromDB(id, userName);
if (result === true) {
// deleteFromDB decide se è soft o hard
// possiamo verificare se la foto esiste ancora nel DB
const stillExists = await db("photos")
.where({ id, user: userName })
.first();
if (stillExists) {
softCount++;
} else {
hardCount++;
}
}
}
// 3) Cancella la cartella thumbs residua (solo se hard delete totale)
const thumbsDir = path.join(
WEB_ROOT,
userName,
"thumbs",
folderName
);
try {
await fsp.rm(thumbsDir, { recursive: true, force: true });
console.log(`🧹 Rimossa cartella thumbs residua: ${thumbsDir}`);
} catch (err) {
console.log(`⚠️ Errore rimozione thumbs dir: ${thumbsDir}`, err);
}
return {
totalPhotos: rows.length,
softDeleted: softCount,
hardDeleted: hardCount,
removedThumbsDir: thumbsDir
};
}
return { deleteFolderForUser };
};

View file

@ -0,0 +1,19 @@
// scanner/deleteWithAuth.js
const { API_KEY } = require('../config');
module.exports = async function deleteWithAuth(url) {
// import dinamico compatibile con CommonJS
const fetch = (await import('node-fetch')).default;
const res = await fetch(url, {
method: 'DELETE',
headers: {
'x-api-key': API_KEY,
'Content-Type': 'application/json'
}
});
if (!res.ok) {
throw new Error(`DELETE failed: ${res.status}`);
}
};

View file

@ -0,0 +1,81 @@
// api_v1/scanner/elevation.js
// ---------------------------------------------------------
// Provider 1: OpenElevation
// ---------------------------------------------------------
async function tryOpenElevation(lat, lon) {
try {
const url = `https://api.open-elevation.com/api/v1/lookup?locations=${lat},${lon}`;
const res = await fetch(url, { timeout: 5000 });
if (!res.ok) {
console.log("OpenElevation status:", res.status);
return null;
}
const text = await res.text();
// Se non è JSON → errore HTML → fallback
if (!text.startsWith('{')) {
console.log("OpenElevation non-JSON:", text.slice(0, 60));
return null;
}
const data = JSON.parse(text);
return data?.results?.[0]?.elevation ?? null;
} catch (err) {
console.log("Errore OpenElevation:", err.message);
return null;
}
}
// ---------------------------------------------------------
// Provider 2: OpenTopoData (fallback)
// ---------------------------------------------------------
async function tryOpenTopoData(lat, lon) {
try {
const url = `https://api.opentopodata.org/v1/eudem25m?locations=${lat},${lon}`;
const res = await fetch(url, { timeout: 5000 });
if (!res.ok) {
console.log("OpenTopoData status:", res.status);
return null;
}
const text = await res.text();
if (!text.startsWith('{')) {
console.log("OpenTopoData non-JSON:", text.slice(0, 60));
return null;
}
const data = JSON.parse(text);
return data?.results?.[0]?.elevation ?? null;
} catch (err) {
console.log("Errore OpenTopoData:", err.message);
return null;
}
}
// ---------------------------------------------------------
// Funzione principale con fallback
// ---------------------------------------------------------
async function getElevation(lat, lon) {
// 1️⃣ Prova OpenElevation
const elev1 = await tryOpenElevation(lat, lon);
if (elev1 !== null) return elev1;
console.log("⚠️ OpenElevation fallito → uso OpenTopoData");
// 2️⃣ Prova OpenTopoData
const elev2 = await tryOpenTopoData(lat, lon);
if (elev2 !== null) return elev2;
console.log("❌ Nessun provider di elevazione disponibile");
return null;
}
module.exports = getElevation;

View file

@ -0,0 +1,395 @@
const db = require('../../db/knex');
const {
scanPhotoFacesById,
} = require('./faceScanner');
function serializeError(error) {
const value =
error?.stack
|| error?.message
|| String(error);
return String(value).slice(0, 16000);
}
async function claimNextPendingJob() {
return db.transaction(async (trx) => {
const job = await trx('face_scan_jobs')
.where({ status: 'pending' })
.orderBy('created_at', 'asc')
.orderBy('photo_id', 'asc')
.first();
if (!job) {
return null;
}
const updated = await trx('face_scan_jobs')
.where({
photo_id: job.photo_id,
status: 'pending',
})
.update({
status: 'processing',
attempts: trx.raw('attempts + 1'),
faces_detected: null,
execution_ms: null,
applied_rotation: null,
last_error: null,
started_at: trx.fn.now(),
completed_at: null,
updated_at: trx.fn.now(),
});
if (updated !== 1) {
return null;
}
return trx('face_scan_jobs')
.where({ photo_id: job.photo_id })
.first();
});
}
async function completeJob(photoId, result) {
return db.transaction(async (trx) => {
const job = await trx('face_scan_jobs')
.where({
photo_id: photoId,
status: 'processing',
})
.first();
if (!job) {
throw new Error(
`Job processing non trovato: photo_id=${photoId}`,
);
}
if (Number(job.rescan_requested) === 1) {
const updated = await trx('face_scan_jobs')
.where({
photo_id: photoId,
status: 'processing',
})
.update({
status: 'pending',
rescan_requested: 0,
faces_detected: null,
execution_ms: null,
applied_rotation: null,
last_error: null,
started_at: null,
completed_at: null,
updated_at: trx.fn.now(),
});
if (updated !== 1) {
throw new Error(
`Impossibile riaccodare il job photo_id=${photoId}`,
);
}
return 'pending';
}
const updated = await trx('face_scan_jobs')
.where({
photo_id: photoId,
status: 'processing',
})
.update({
status: 'done',
rescan_requested: 0,
faces_detected: result.faces_detected,
execution_ms: result.execution_ms,
applied_rotation: result.applied_rotation,
last_error: null,
completed_at: trx.fn.now(),
updated_at: trx.fn.now(),
});
if (updated !== 1) {
throw new Error(
`Impossibile completare il job photo_id=${photoId}`,
);
}
return 'done';
});
}
async function failJob(photoId, error) {
return db.transaction(async (trx) => {
const job = await trx('face_scan_jobs')
.where({
photo_id: photoId,
status: 'processing',
})
.first();
if (!job) {
console.error(
'[faceScanWorker] job processing non trovato:',
photoId,
);
return 'missing';
}
if (Number(job.rescan_requested) === 1) {
const updated = await trx('face_scan_jobs')
.where({
photo_id: photoId,
status: 'processing',
})
.update({
status: 'pending',
rescan_requested: 0,
faces_detected: null,
execution_ms: null,
applied_rotation: null,
last_error: null,
started_at: null,
completed_at: null,
updated_at: trx.fn.now(),
});
if (updated !== 1) {
throw new Error(
`Impossibile riaccodare il job photo_id=${photoId}`,
);
}
return 'pending';
}
const updated = await trx('face_scan_jobs')
.where({
photo_id: photoId,
status: 'processing',
})
.update({
status: 'error',
rescan_requested: 0,
last_error: serializeError(error),
completed_at: trx.fn.now(),
updated_at: trx.fn.now(),
});
if (updated !== 1) {
console.error(
'[faceScanWorker] impossibile registrare errore:',
photoId,
);
return 'missing';
}
return 'error';
});
}
async function processClaimedJob(job) {
if (!job || !job.photo_id) {
throw new Error('Job non valido');
}
const workerStartedAt = Date.now();
console.log(
'[faceScanWorker] processing',
{
photo_id: job.photo_id,
attempt: job.attempts,
},
);
try {
const result = await scanPhotoFacesById(
job.photo_id,
);
const finalStatus = await completeJob(
job.photo_id,
result,
);
const summary = {
status: finalStatus,
photo_id: job.photo_id,
attempt: job.attempts,
faces_detected:
finalStatus === 'done'
? result.faces_detected
: null,
applied_rotation:
finalStatus === 'done'
? result.applied_rotation
: null,
execution_ms:
finalStatus === 'done'
? result.execution_ms
: null,
rescan_required:
finalStatus === 'pending',
worker_ms: Date.now() - workerStartedAt,
};
console.log(
'[faceScanWorker] done',
summary,
);
return summary;
} catch (error) {
const finalStatus = await failJob(
job.photo_id,
error,
);
const summary = {
status: finalStatus,
photo_id: job.photo_id,
attempt: job.attempts,
error:
finalStatus === 'error'
? serializeError(error)
: null,
rescan_required:
finalStatus === 'pending',
worker_ms: Date.now() - workerStartedAt,
};
console.error(
'[faceScanWorker] error',
summary,
);
return summary;
}
}
async function processNextJob() {
const job = await claimNextPendingJob();
if (!job) {
return null;
}
return processClaimedJob(job);
}
async function runWorker(options = {}) {
const maxJobs =
options.maxJobs == null
? null
: Number(options.maxJobs);
if (
maxJobs !== null
&& (
!Number.isInteger(maxJobs)
|| maxJobs <= 0
)
) {
throw new Error(
'maxJobs deve essere un intero positivo',
);
}
const shouldStop =
typeof options.shouldStop === 'function'
? options.shouldStop
: () => false;
const results = [];
while (
!shouldStop()
&& (
maxJobs === null
|| results.length < maxJobs
)
) {
const result = await processNextJob();
if (!result) {
break;
}
results.push(result);
}
const succeeded = results.filter(
(result) => result.status === 'done',
).length;
const failed = results.filter(
(result) => result.status === 'error',
).length;
const rescheduled = results.filter(
(result) => result.status === 'pending',
).length;
return {
jobs_processed: results.length,
jobs_succeeded: succeeded,
jobs_failed: failed,
jobs_rescheduled: rescheduled,
stopped: shouldStop(),
results,
};
}
async function resetStaleProcessingJobs(
staleMinutes = 30,
) {
const minutes = Number(staleMinutes);
if (
!Number.isFinite(minutes)
|| minutes <= 0
) {
throw new Error(
'staleMinutes deve essere positivo',
);
}
const modifier =
`-${Math.floor(minutes)} minutes`;
const updated = await db('face_scan_jobs')
.where({ status: 'processing' })
.where(
'started_at',
'<',
db.raw(
"datetime('now', ?)",
[modifier],
),
)
.update({
status: 'pending',
last_error:
'Job recuperato dopo arresto del worker',
started_at: null,
completed_at: null,
updated_at: db.fn.now(),
});
return updated;
}
module.exports = {
serializeError,
claimNextPendingJob,
completeJob,
failJob,
processClaimedJob,
processNextJob,
runWorker,
resetStaleProcessingJobs,
};

View file

@ -0,0 +1,279 @@
const db = require('../../db/knex');
const {
scanPhotoFacesById,
} = require('./faceScanner');
function serializeError(error) {
const value =
error?.stack
|| error?.message
|| String(error);
return String(value).slice(0, 16000);
}
async function claimNextPendingJob() {
return db.transaction(async (trx) => {
const job = await trx('face_scan_jobs')
.where({ status: 'pending' })
.orderBy('created_at', 'asc')
.orderBy('photo_id', 'asc')
.first();
if (!job) {
return null;
}
const updated = await trx('face_scan_jobs')
.where({
photo_id: job.photo_id,
status: 'pending',
})
.update({
status: 'processing',
attempts: trx.raw('attempts + 1'),
faces_detected: null,
execution_ms: null,
applied_rotation: null,
last_error: null,
started_at: trx.fn.now(),
completed_at: null,
updated_at: trx.fn.now(),
});
if (updated !== 1) {
return null;
}
return trx('face_scan_jobs')
.where({ photo_id: job.photo_id })
.first();
});
}
async function completeJob(photoId, result) {
const updated = await db('face_scan_jobs')
.where({
photo_id: photoId,
status: 'processing',
})
.update({
status: 'done',
faces_detected: result.faces_detected,
execution_ms: result.execution_ms,
applied_rotation: result.applied_rotation,
last_error: null,
completed_at: db.fn.now(),
updated_at: db.fn.now(),
});
if (updated !== 1) {
throw new Error(
`Impossibile completare il job photo_id=${photoId}`,
);
}
}
async function failJob(photoId, error) {
const updated = await db('face_scan_jobs')
.where({
photo_id: photoId,
status: 'processing',
})
.update({
status: 'error',
last_error: serializeError(error),
completed_at: db.fn.now(),
updated_at: db.fn.now(),
});
if (updated !== 1) {
console.error(
'[faceScanWorker] impossibile registrare errore:',
photoId,
);
}
}
async function processClaimedJob(job) {
if (!job || !job.photo_id) {
throw new Error('Job non valido');
}
const workerStartedAt = Date.now();
console.log(
'[faceScanWorker] processing',
{
photo_id: job.photo_id,
attempt: job.attempts,
},
);
try {
const result = await scanPhotoFacesById(
job.photo_id,
);
await completeJob(
job.photo_id,
result,
);
const summary = {
status: 'done',
photo_id: job.photo_id,
attempt: job.attempts,
faces_detected: result.faces_detected,
applied_rotation: result.applied_rotation,
execution_ms: result.execution_ms,
worker_ms: Date.now() - workerStartedAt,
};
console.log(
'[faceScanWorker] done',
summary,
);
return summary;
} catch (error) {
await failJob(
job.photo_id,
error,
);
const summary = {
status: 'error',
photo_id: job.photo_id,
attempt: job.attempts,
error: serializeError(error),
worker_ms: Date.now() - workerStartedAt,
};
console.error(
'[faceScanWorker] error',
summary,
);
return summary;
}
}
async function processNextJob() {
const job = await claimNextPendingJob();
if (!job) {
return null;
}
return processClaimedJob(job);
}
async function runWorker(options = {}) {
const maxJobs =
options.maxJobs == null
? null
: Number(options.maxJobs);
if (
maxJobs !== null
&& (
!Number.isInteger(maxJobs)
|| maxJobs <= 0
)
) {
throw new Error(
'maxJobs deve essere un intero positivo',
);
}
const shouldStop =
typeof options.shouldStop === 'function'
? options.shouldStop
: () => false;
const results = [];
while (
!shouldStop()
&& (
maxJobs === null
|| results.length < maxJobs
)
) {
const result = await processNextJob();
if (!result) {
break;
}
results.push(result);
}
const succeeded = results.filter(
(result) => result.status === 'done',
).length;
const failed = results.filter(
(result) => result.status === 'error',
).length;
return {
jobs_processed: results.length,
jobs_succeeded: succeeded,
jobs_failed: failed,
stopped: shouldStop(),
results,
};
}
async function resetStaleProcessingJobs(
staleMinutes = 30,
) {
const minutes = Number(staleMinutes);
if (
!Number.isFinite(minutes)
|| minutes <= 0
) {
throw new Error(
'staleMinutes deve essere positivo',
);
}
const modifier =
`-${Math.floor(minutes)} minutes`;
const updated = await db('face_scan_jobs')
.where({ status: 'processing' })
.where(
'started_at',
'<',
db.raw(
"datetime('now', ?)",
[modifier],
),
)
.update({
status: 'pending',
last_error:
'Job recuperato dopo arresto del worker',
started_at: null,
completed_at: null,
updated_at: db.fn.now(),
});
return updated;
}
module.exports = {
serializeError,
claimNextPendingJob,
completeJob,
failJob,
processClaimedJob,
processNextJob,
runWorker,
resetStaleProcessingJobs,
};

View file

@ -0,0 +1,519 @@
const fs = require('fs');
const path = require('path');
const { execFile } = require('child_process');
const { promisify } = require('util');
const db = require('../../db/knex');
const {
resolveFilePathForEntryId,
} = require('../../utils/resolveFilePath');
const execFileAsync = promisify(execFile);
const EMBEDDING_DIM = 128;
const EMBEDDING_BYTES = EMBEDDING_DIM * 4;
const PROJECT_ROOT =
path.resolve(__dirname, '../..');
function getFaceScanBinary() {
if (process.env.FACE_SCAN_BINARY) {
return path.resolve(
process.env.FACE_SCAN_BINARY,
);
}
return path.join(
PROJECT_ROOT,
'bin',
'face_scan',
);
}
function isFiniteNumber(value) {
return (
typeof value === 'number'
&& Number.isFinite(value)
);
}
function validateImage(image) {
if (!image || typeof image !== 'object') {
throw new Error(
'Risultato face_scan non valido: image mancante',
);
}
if (
!Number.isInteger(image.width)
|| image.width <= 0
) {
throw new Error(
`Larghezza immagine non valida: ${image.width}`,
);
}
if (
!Number.isInteger(image.height)
|| image.height <= 0
) {
throw new Error(
`Altezza immagine non valida: ${image.height}`,
);
}
if (typeof image.path !== 'string') {
throw new Error(
'Percorso immagine non valido nel risultato face_scan',
);
}
}
function validateBoundingBox(bbox, faceIndex) {
if (!bbox || typeof bbox !== 'object') {
throw new Error(
`Bounding box mancante per face_index=${faceIndex}`,
);
}
for (const key of ['x', 'y', 'width', 'height']) {
if (!isFiniteNumber(bbox[key])) {
throw new Error(
`bbox.${key} non valido per face_index=${faceIndex}`,
);
}
}
if (bbox.width < 0 || bbox.height < 0) {
throw new Error(
`Bounding box negativa per face_index=${faceIndex}`,
);
}
}
function validateLandmarks(landmarks, faceIndex) {
if (!Array.isArray(landmarks)) {
throw new Error(
`Landmark non validi per face_index=${faceIndex}`,
);
}
if (landmarks.length !== 5) {
throw new Error(
`Landmark non validi per face_index=${faceIndex}: `
+ `attesi 5, ricevuti ${landmarks.length}`,
);
}
landmarks.forEach((point, landmarkIndex) => {
if (
!Array.isArray(point)
|| point.length !== 2
|| !isFiniteNumber(point[0])
|| !isFiniteNumber(point[1])
) {
throw new Error(
`Landmark ${landmarkIndex} non valido `
+ `per face_index=${faceIndex}`,
);
}
});
}
function validateEmbedding(embedding, faceIndex) {
if (!Array.isArray(embedding)) {
throw new Error(
`Embedding non valido per face_index=${faceIndex}`,
);
}
if (embedding.length !== EMBEDDING_DIM) {
throw new Error(
`Embedding non valido per face_index=${faceIndex}: `
+ `attesi ${EMBEDDING_DIM}, `
+ `ricevuti ${embedding.length}`,
);
}
for (
let embeddingIndex = 0;
embeddingIndex < embedding.length;
embeddingIndex += 1
) {
if (!isFiniteNumber(embedding[embeddingIndex])) {
throw new Error(
`Embedding non finito per face_index=${faceIndex}, `
+ `indice=${embeddingIndex}`,
);
}
}
const normSquared = embedding.reduce(
(sum, value) => sum + (value * value),
0,
);
const norm = Math.sqrt(normSquared);
if (
!Number.isFinite(norm)
|| Math.abs(norm - 1) > 0.001
) {
throw new Error(
`Embedding non normalizzato per face_index=${faceIndex}: `
+ `norm=${norm}`,
);
}
}
function validateFace(face, expectedIndex) {
if (!face || typeof face !== 'object') {
throw new Error(
`Volto non valido all'indice ${expectedIndex}`,
);
}
if (
!Number.isInteger(face.face_index)
|| face.face_index !== expectedIndex
) {
throw new Error(
`face_index non valido: atteso ${expectedIndex}, `
+ `ricevuto ${face.face_index}`,
);
}
if (!isFiniteNumber(face.score)) {
throw new Error(
`Score non valido per face_index=${expectedIndex}`,
);
}
validateBoundingBox(face.bbox, expectedIndex);
validateLandmarks(face.landmarks, expectedIndex);
validateEmbedding(face.embedding, expectedIndex);
}
function validateFaceScanResult(result) {
if (!result || typeof result !== 'object') {
throw new Error(
'Risultato face_scan non valido',
);
}
validateImage(result.image);
if (!Array.isArray(result.faces)) {
throw new Error(
'Risultato face_scan non valido: faces non è un array',
);
}
result.faces.forEach((face, faceIndex) => {
validateFace(face, faceIndex);
});
return result;
}
function embeddingToBlob(embedding) {
if (
!Array.isArray(embedding)
|| embedding.length !== EMBEDDING_DIM
) {
throw new Error(
`Embedding non valido: attesi ${EMBEDDING_DIM} valori`,
);
}
const blob = Buffer.allocUnsafe(EMBEDDING_BYTES);
embedding.forEach((value, index) => {
if (!isFiniteNumber(value)) {
throw new Error(
`Valore embedding non valido all'indice ${index}`,
);
}
blob.writeFloatLE(value, index * 4);
});
if (blob.length !== EMBEDDING_BYTES) {
throw new Error(
`Dimensione BLOB non valida: ${blob.length}`,
);
}
return blob;
}
async function verifyFaceScanBinary() {
const binaryPath = getFaceScanBinary();
try {
await fs.promises.access(
binaryPath,
fs.constants.X_OK,
);
} catch {
throw new Error(
`face_scan non trovato o non eseguibile: ${binaryPath}`,
);
}
return binaryPath;
}
async function runFaceScan(
imagePath,
rotationDegrees = 0,
) {
const binaryPath = await verifyFaceScanBinary();
const normalizedRotation =
Number(rotationDegrees);
if (
!Number.isInteger(normalizedRotation)
|| ![0, 90, 180, 270].includes(normalizedRotation)
) {
throw new Error(
`Rotazione foto non valida: ${rotationDegrees}`,
);
}
const startedAt = Date.now();
let stdout;
let stderr;
try {
const result = await execFileAsync(
binaryPath,
[
'--rotation',
String(normalizedRotation),
imagePath,
],
{
encoding: 'utf8',
timeout: Number(
process.env.FACE_SCAN_TIMEOUT_MS || 300000,
),
maxBuffer: Number(
process.env.FACE_SCAN_MAX_BUFFER
|| 64 * 1024 * 1024,
),
windowsHide: true,
},
);
stdout = result.stdout;
stderr = result.stderr;
} catch (error) {
const diagnostics = [
error.message,
error.stderr,
]
.filter(Boolean)
.join('\n')
.trim();
throw new Error(
`face_scan fallito per ${imagePath}`
+ (diagnostics ? `\n${diagnostics}` : ''),
);
}
if (stderr && stderr.trim()) {
console.warn(
'[faceScanner] face_scan stderr:',
stderr.trim(),
);
}
if (!stdout || !stdout.trim()) {
throw new Error(
`face_scan non ha prodotto JSON per ${imagePath}`,
);
}
let parsed;
try {
parsed = JSON.parse(stdout);
} catch (error) {
throw new Error(
`JSON face_scan non valido per ${imagePath}: `
+ error.message,
);
}
validateFaceScanResult(parsed);
if (
parsed.image.rotation !== normalizedRotation
) {
throw new Error(
`Rotazione face_scan inattesa: richiesta `
+ `${normalizedRotation}, ricevuta `
+ `${parsed.image.rotation}`,
);
}
return {
...parsed,
execution_ms: Date.now() - startedAt,
};
}
function faceToDatabaseRow(photoId, face) {
const landmarks = face.landmarks;
return {
id: `face:${photoId}:${face.face_index}`,
photo_id: photoId,
face_index: face.face_index,
x: face.bbox.x,
y: face.bbox.y,
width: face.bbox.width,
height: face.bbox.height,
score: face.score,
landmark_0_x: landmarks[0][0],
landmark_0_y: landmarks[0][1],
landmark_1_x: landmarks[1][0],
landmark_1_y: landmarks[1][1],
landmark_2_x: landmarks[2][0],
landmark_2_y: landmarks[2][1],
landmark_3_x: landmarks[3][0],
landmark_3_y: landmarks[3][1],
landmark_4_x: landmarks[4][0],
landmark_4_y: landmarks[4][1],
embedding: embeddingToBlob(face.embedding),
cluster_id: null,
person_id: null,
status: 'detected',
created_at: db.fn.now(),
updated_at: db.fn.now(),
};
}
async function replacePhotoFaces(photoId, faces) {
const rows = faces.map((face) => (
faceToDatabaseRow(photoId, face)
));
await db.transaction(async (trx) => {
await trx('faces')
.where({ photo_id: photoId })
.del();
if (rows.length > 0) {
await trx('faces').insert(rows);
}
});
return rows.length;
}
async function scanPhotoFacesById(photoId) {
const cleanPhotoId = String(photoId || '').trim();
if (!cleanPhotoId) {
throw new Error('photoId mancante');
}
const photo = await db('photos')
.where({ id: cleanPhotoId })
.whereNull('deleted_at')
.first();
if (!photo) {
throw new Error(
`Foto non trovata o eliminata: ${cleanPhotoId}`,
);
}
if (
photo.mime_type
&& !String(photo.mime_type).startsWith('image/')
) {
throw new Error(
`Il record non è un'immagine: `
+ `${photo.mime_type}`,
);
}
const imagePath =
await resolveFilePathForEntryId(cleanPhotoId);
if (!imagePath) {
throw new Error(
`Percorso fisico non risolto per photo_id=${cleanPhotoId}`,
);
}
let fileStat;
try {
fileStat = await fs.promises.stat(imagePath);
} catch (error) {
throw new Error(
`File non accessibile: ${imagePath}: ${error.message}`,
);
}
if (!fileStat.isFile()) {
throw new Error(
`Il percorso non è un file: ${imagePath}`,
);
}
const rotationDegrees =
photo.rotation == null
? 0
: Number(photo.rotation);
const result = await runFaceScan(
imagePath,
rotationDegrees,
);
const insertedFaces = await replacePhotoFaces(
cleanPhotoId,
result.faces,
);
return {
photo_id: cleanPhotoId,
image_path: imagePath,
image_width: result.image.width,
image_height: result.image.height,
photo_rotation: photo.rotation ?? null,
applied_rotation: result.image.rotation,
faces_detected: result.faces.length,
faces_inserted: insertedFaces,
execution_ms: result.execution_ms,
};
}
module.exports = {
EMBEDDING_DIM,
EMBEDDING_BYTES,
embeddingToBlob,
validateFaceScanResult,
runFaceScan,
replacePhotoFaces,
scanPhotoFacesById,
};

View file

@ -0,0 +1,486 @@
const fs = require('fs');
const path = require('path');
const { execFile } = require('child_process');
const { promisify } = require('util');
const db = require('../../db/knex');
const {
resolveFilePathForEntryId,
} = require('../../utils/resolveFilePath');
const execFileAsync = promisify(execFile);
const EMBEDDING_DIM = 128;
const EMBEDDING_BYTES = EMBEDDING_DIM * 4;
//const DEFAULT_FACE_CLUSTERING_ROOT =
// '/home/nvme/dockerdata/prove/face-clustering-rk3588';
function getFaceClusteringRoot() {
return path.resolve(
process.env.FACE_CLUSTERING_ROOT
|| DEFAULT_FACE_CLUSTERING_ROOT,
);
}
function getFaceScanBinary() {
if (process.env.FACE_SCAN_BINARY) {
return path.resolve(process.env.FACE_SCAN_BINARY);
}
return path.join(
getFaceClusteringRoot(),
'bin',
'face_scan',
);
}
function isFiniteNumber(value) {
return (
typeof value === 'number'
&& Number.isFinite(value)
);
}
function validateImage(image) {
if (!image || typeof image !== 'object') {
throw new Error(
'Risultato face_scan non valido: image mancante',
);
}
if (
!Number.isInteger(image.width)
|| image.width <= 0
) {
throw new Error(
`Larghezza immagine non valida: ${image.width}`,
);
}
if (
!Number.isInteger(image.height)
|| image.height <= 0
) {
throw new Error(
`Altezza immagine non valida: ${image.height}`,
);
}
if (typeof image.path !== 'string') {
throw new Error(
'Percorso immagine non valido nel risultato face_scan',
);
}
}
function validateBoundingBox(bbox, faceIndex) {
if (!bbox || typeof bbox !== 'object') {
throw new Error(
`Bounding box mancante per face_index=${faceIndex}`,
);
}
for (const key of ['x', 'y', 'width', 'height']) {
if (!isFiniteNumber(bbox[key])) {
throw new Error(
`bbox.${key} non valido per face_index=${faceIndex}`,
);
}
}
if (bbox.width < 0 || bbox.height < 0) {
throw new Error(
`Bounding box negativa per face_index=${faceIndex}`,
);
}
}
function validateLandmarks(landmarks, faceIndex) {
if (!Array.isArray(landmarks)) {
throw new Error(
`Landmark non validi per face_index=${faceIndex}`,
);
}
if (landmarks.length !== 5) {
throw new Error(
`Landmark non validi per face_index=${faceIndex}: `
+ `attesi 5, ricevuti ${landmarks.length}`,
);
}
landmarks.forEach((point, landmarkIndex) => {
if (
!Array.isArray(point)
|| point.length !== 2
|| !isFiniteNumber(point[0])
|| !isFiniteNumber(point[1])
) {
throw new Error(
`Landmark ${landmarkIndex} non valido `
+ `per face_index=${faceIndex}`,
);
}
});
}
function validateEmbedding(embedding, faceIndex) {
if (!Array.isArray(embedding)) {
throw new Error(
`Embedding non valido per face_index=${faceIndex}`,
);
}
if (embedding.length !== EMBEDDING_DIM) {
throw new Error(
`Embedding non valido per face_index=${faceIndex}: `
+ `attesi ${EMBEDDING_DIM}, `
+ `ricevuti ${embedding.length}`,
);
}
for (
let embeddingIndex = 0;
embeddingIndex < embedding.length;
embeddingIndex += 1
) {
if (!isFiniteNumber(embedding[embeddingIndex])) {
throw new Error(
`Embedding non finito per face_index=${faceIndex}, `
+ `indice=${embeddingIndex}`,
);
}
}
const normSquared = embedding.reduce(
(sum, value) => sum + (value * value),
0,
);
const norm = Math.sqrt(normSquared);
if (
!Number.isFinite(norm)
|| Math.abs(norm - 1) > 0.001
) {
throw new Error(
`Embedding non normalizzato per face_index=${faceIndex}: `
+ `norm=${norm}`,
);
}
}
function validateFace(face, expectedIndex) {
if (!face || typeof face !== 'object') {
throw new Error(
`Volto non valido all'indice ${expectedIndex}`,
);
}
if (
!Number.isInteger(face.face_index)
|| face.face_index !== expectedIndex
) {
throw new Error(
`face_index non valido: atteso ${expectedIndex}, `
+ `ricevuto ${face.face_index}`,
);
}
if (!isFiniteNumber(face.score)) {
throw new Error(
`Score non valido per face_index=${expectedIndex}`,
);
}
validateBoundingBox(face.bbox, expectedIndex);
validateLandmarks(face.landmarks, expectedIndex);
validateEmbedding(face.embedding, expectedIndex);
}
function validateFaceScanResult(result) {
if (!result || typeof result !== 'object') {
throw new Error(
'Risultato face_scan non valido',
);
}
validateImage(result.image);
if (!Array.isArray(result.faces)) {
throw new Error(
'Risultato face_scan non valido: faces non è un array',
);
}
result.faces.forEach((face, faceIndex) => {
validateFace(face, faceIndex);
});
return result;
}
function embeddingToBlob(embedding) {
if (
!Array.isArray(embedding)
|| embedding.length !== EMBEDDING_DIM
) {
throw new Error(
`Embedding non valido: attesi ${EMBEDDING_DIM} valori`,
);
}
const blob = Buffer.allocUnsafe(EMBEDDING_BYTES);
embedding.forEach((value, index) => {
if (!isFiniteNumber(value)) {
throw new Error(
`Valore embedding non valido all'indice ${index}`,
);
}
blob.writeFloatLE(value, index * 4);
});
if (blob.length !== EMBEDDING_BYTES) {
throw new Error(
`Dimensione BLOB non valida: ${blob.length}`,
);
}
return blob;
}
async function verifyFaceScanBinary() {
const binaryPath = getFaceScanBinary();
try {
await fs.promises.access(
binaryPath,
fs.constants.X_OK,
);
} catch {
throw new Error(
`face_scan non trovato o non eseguibile: ${binaryPath}`,
);
}
return binaryPath;
}
async function runFaceScan(imagePath) {
const binaryPath = await verifyFaceScanBinary();
const startedAt = Date.now();
let stdout;
let stderr;
try {
const result = await execFileAsync(
binaryPath,
[imagePath],
{
encoding: 'utf8',
timeout: Number(
process.env.FACE_SCAN_TIMEOUT_MS || 300000,
),
maxBuffer: Number(
process.env.FACE_SCAN_MAX_BUFFER
|| 64 * 1024 * 1024,
),
windowsHide: true,
},
);
stdout = result.stdout;
stderr = result.stderr;
} catch (error) {
const diagnostics = [
error.message,
error.stderr,
]
.filter(Boolean)
.join('\n')
.trim();
throw new Error(
`face_scan fallito per ${imagePath}`
+ (diagnostics ? `\n${diagnostics}` : ''),
);
}
if (stderr && stderr.trim()) {
console.warn(
'[faceScanner] face_scan stderr:',
stderr.trim(),
);
}
if (!stdout || !stdout.trim()) {
throw new Error(
`face_scan non ha prodotto JSON per ${imagePath}`,
);
}
let parsed;
try {
parsed = JSON.parse(stdout);
} catch (error) {
throw new Error(
`JSON face_scan non valido per ${imagePath}: `
+ error.message,
);
}
validateFaceScanResult(parsed);
return {
...parsed,
execution_ms: Date.now() - startedAt,
};
}
function faceToDatabaseRow(photoId, face) {
const landmarks = face.landmarks;
return {
id: `face:${photoId}:${face.face_index}`,
photo_id: photoId,
face_index: face.face_index,
x: face.bbox.x,
y: face.bbox.y,
width: face.bbox.width,
height: face.bbox.height,
score: face.score,
landmark_0_x: landmarks[0][0],
landmark_0_y: landmarks[0][1],
landmark_1_x: landmarks[1][0],
landmark_1_y: landmarks[1][1],
landmark_2_x: landmarks[2][0],
landmark_2_y: landmarks[2][1],
landmark_3_x: landmarks[3][0],
landmark_3_y: landmarks[3][1],
landmark_4_x: landmarks[4][0],
landmark_4_y: landmarks[4][1],
embedding: embeddingToBlob(face.embedding),
cluster_id: null,
person_id: null,
status: 'detected',
created_at: db.fn.now(),
updated_at: db.fn.now(),
};
}
async function replacePhotoFaces(photoId, faces) {
const rows = faces.map((face) => (
faceToDatabaseRow(photoId, face)
));
await db.transaction(async (trx) => {
await trx('faces')
.where({ photo_id: photoId })
.del();
if (rows.length > 0) {
await trx('faces').insert(rows);
}
});
return rows.length;
}
async function scanPhotoFacesById(photoId) {
const cleanPhotoId = String(photoId || '').trim();
if (!cleanPhotoId) {
throw new Error('photoId mancante');
}
const photo = await db('photos')
.where({ id: cleanPhotoId })
.whereNull('deleted_at')
.first();
if (!photo) {
throw new Error(
`Foto non trovata o eliminata: ${cleanPhotoId}`,
);
}
if (
photo.mime_type
&& !String(photo.mime_type).startsWith('image/')
) {
throw new Error(
`Il record non è un'immagine: `
+ `${photo.mime_type}`,
);
}
const imagePath =
await resolveFilePathForEntryId(cleanPhotoId);
if (!imagePath) {
throw new Error(
`Percorso fisico non risolto per photo_id=${cleanPhotoId}`,
);
}
let fileStat;
try {
fileStat = await fs.promises.stat(imagePath);
} catch (error) {
throw new Error(
`File non accessibile: ${imagePath}: ${error.message}`,
);
}
if (!fileStat.isFile()) {
throw new Error(
`Il percorso non è un file: ${imagePath}`,
);
}
const result = await runFaceScan(imagePath);
const insertedFaces = await replacePhotoFaces(
cleanPhotoId,
result.faces,
);
return {
photo_id: cleanPhotoId,
image_path: imagePath,
image_width: result.image.width,
image_height: result.image.height,
photo_rotation: photo.rotation ?? null,
faces_detected: result.faces.length,
faces_inserted: insertedFaces,
execution_ms: result.execution_ms,
};
}
module.exports = {
EMBEDDING_DIM,
EMBEDDING_BYTES,
embeddingToBlob,
validateFaceScanResult,
runFaceScan,
replacePhotoFaces,
scanPhotoFacesById,
};

View file

@ -0,0 +1,481 @@
const fs = require('fs');
const path = require('path');
const { execFile } = require('child_process');
const { promisify } = require('util');
const db = require('../../db/knex');
const {
resolveFilePathForEntryId,
} = require('../../utils/resolveFilePath');
const execFileAsync = promisify(execFile);
const EMBEDDING_DIM = 128;
const EMBEDDING_BYTES = EMBEDDING_DIM * 4;
const PROJECT_ROOT =
path.resolve(__dirname, '../..');
function getFaceScanBinary() {
if (process.env.FACE_SCAN_BINARY) {
return path.resolve(
process.env.FACE_SCAN_BINARY,
);
}
return path.join(
PROJECT_ROOT,
'bin',
'face_scan',
);
}
function isFiniteNumber(value) {
return (
typeof value === 'number'
&& Number.isFinite(value)
);
}
function validateImage(image) {
if (!image || typeof image !== 'object') {
throw new Error(
'Risultato face_scan non valido: image mancante',
);
}
if (
!Number.isInteger(image.width)
|| image.width <= 0
) {
throw new Error(
`Larghezza immagine non valida: ${image.width}`,
);
}
if (
!Number.isInteger(image.height)
|| image.height <= 0
) {
throw new Error(
`Altezza immagine non valida: ${image.height}`,
);
}
if (typeof image.path !== 'string') {
throw new Error(
'Percorso immagine non valido nel risultato face_scan',
);
}
}
function validateBoundingBox(bbox, faceIndex) {
if (!bbox || typeof bbox !== 'object') {
throw new Error(
`Bounding box mancante per face_index=${faceIndex}`,
);
}
for (const key of ['x', 'y', 'width', 'height']) {
if (!isFiniteNumber(bbox[key])) {
throw new Error(
`bbox.${key} non valido per face_index=${faceIndex}`,
);
}
}
if (bbox.width < 0 || bbox.height < 0) {
throw new Error(
`Bounding box negativa per face_index=${faceIndex}`,
);
}
}
function validateLandmarks(landmarks, faceIndex) {
if (!Array.isArray(landmarks)) {
throw new Error(
`Landmark non validi per face_index=${faceIndex}`,
);
}
if (landmarks.length !== 5) {
throw new Error(
`Landmark non validi per face_index=${faceIndex}: `
+ `attesi 5, ricevuti ${landmarks.length}`,
);
}
landmarks.forEach((point, landmarkIndex) => {
if (
!Array.isArray(point)
|| point.length !== 2
|| !isFiniteNumber(point[0])
|| !isFiniteNumber(point[1])
) {
throw new Error(
`Landmark ${landmarkIndex} non valido `
+ `per face_index=${faceIndex}`,
);
}
});
}
function validateEmbedding(embedding, faceIndex) {
if (!Array.isArray(embedding)) {
throw new Error(
`Embedding non valido per face_index=${faceIndex}`,
);
}
if (embedding.length !== EMBEDDING_DIM) {
throw new Error(
`Embedding non valido per face_index=${faceIndex}: `
+ `attesi ${EMBEDDING_DIM}, `
+ `ricevuti ${embedding.length}`,
);
}
for (
let embeddingIndex = 0;
embeddingIndex < embedding.length;
embeddingIndex += 1
) {
if (!isFiniteNumber(embedding[embeddingIndex])) {
throw new Error(
`Embedding non finito per face_index=${faceIndex}, `
+ `indice=${embeddingIndex}`,
);
}
}
const normSquared = embedding.reduce(
(sum, value) => sum + (value * value),
0,
);
const norm = Math.sqrt(normSquared);
if (
!Number.isFinite(norm)
|| Math.abs(norm - 1) > 0.001
) {
throw new Error(
`Embedding non normalizzato per face_index=${faceIndex}: `
+ `norm=${norm}`,
);
}
}
function validateFace(face, expectedIndex) {
if (!face || typeof face !== 'object') {
throw new Error(
`Volto non valido all'indice ${expectedIndex}`,
);
}
if (
!Number.isInteger(face.face_index)
|| face.face_index !== expectedIndex
) {
throw new Error(
`face_index non valido: atteso ${expectedIndex}, `
+ `ricevuto ${face.face_index}`,
);
}
if (!isFiniteNumber(face.score)) {
throw new Error(
`Score non valido per face_index=${expectedIndex}`,
);
}
validateBoundingBox(face.bbox, expectedIndex);
validateLandmarks(face.landmarks, expectedIndex);
validateEmbedding(face.embedding, expectedIndex);
}
function validateFaceScanResult(result) {
if (!result || typeof result !== 'object') {
throw new Error(
'Risultato face_scan non valido',
);
}
validateImage(result.image);
if (!Array.isArray(result.faces)) {
throw new Error(
'Risultato face_scan non valido: faces non è un array',
);
}
result.faces.forEach((face, faceIndex) => {
validateFace(face, faceIndex);
});
return result;
}
function embeddingToBlob(embedding) {
if (
!Array.isArray(embedding)
|| embedding.length !== EMBEDDING_DIM
) {
throw new Error(
`Embedding non valido: attesi ${EMBEDDING_DIM} valori`,
);
}
const blob = Buffer.allocUnsafe(EMBEDDING_BYTES);
embedding.forEach((value, index) => {
if (!isFiniteNumber(value)) {
throw new Error(
`Valore embedding non valido all'indice ${index}`,
);
}
blob.writeFloatLE(value, index * 4);
});
if (blob.length !== EMBEDDING_BYTES) {
throw new Error(
`Dimensione BLOB non valida: ${blob.length}`,
);
}
return blob;
}
async function verifyFaceScanBinary() {
const binaryPath = getFaceScanBinary();
try {
await fs.promises.access(
binaryPath,
fs.constants.X_OK,
);
} catch {
throw new Error(
`face_scan non trovato o non eseguibile: ${binaryPath}`,
);
}
return binaryPath;
}
async function runFaceScan(imagePath) {
const binaryPath = await verifyFaceScanBinary();
const startedAt = Date.now();
let stdout;
let stderr;
try {
const result = await execFileAsync(
binaryPath,
[imagePath],
{
encoding: 'utf8',
timeout: Number(
process.env.FACE_SCAN_TIMEOUT_MS || 300000,
),
maxBuffer: Number(
process.env.FACE_SCAN_MAX_BUFFER
|| 64 * 1024 * 1024,
),
windowsHide: true,
},
);
stdout = result.stdout;
stderr = result.stderr;
} catch (error) {
const diagnostics = [
error.message,
error.stderr,
]
.filter(Boolean)
.join('\n')
.trim();
throw new Error(
`face_scan fallito per ${imagePath}`
+ (diagnostics ? `\n${diagnostics}` : ''),
);
}
if (stderr && stderr.trim()) {
console.warn(
'[faceScanner] face_scan stderr:',
stderr.trim(),
);
}
if (!stdout || !stdout.trim()) {
throw new Error(
`face_scan non ha prodotto JSON per ${imagePath}`,
);
}
let parsed;
try {
parsed = JSON.parse(stdout);
} catch (error) {
throw new Error(
`JSON face_scan non valido per ${imagePath}: `
+ error.message,
);
}
validateFaceScanResult(parsed);
return {
...parsed,
execution_ms: Date.now() - startedAt,
};
}
function faceToDatabaseRow(photoId, face) {
const landmarks = face.landmarks;
return {
id: `face:${photoId}:${face.face_index}`,
photo_id: photoId,
face_index: face.face_index,
x: face.bbox.x,
y: face.bbox.y,
width: face.bbox.width,
height: face.bbox.height,
score: face.score,
landmark_0_x: landmarks[0][0],
landmark_0_y: landmarks[0][1],
landmark_1_x: landmarks[1][0],
landmark_1_y: landmarks[1][1],
landmark_2_x: landmarks[2][0],
landmark_2_y: landmarks[2][1],
landmark_3_x: landmarks[3][0],
landmark_3_y: landmarks[3][1],
landmark_4_x: landmarks[4][0],
landmark_4_y: landmarks[4][1],
embedding: embeddingToBlob(face.embedding),
cluster_id: null,
person_id: null,
status: 'detected',
created_at: db.fn.now(),
updated_at: db.fn.now(),
};
}
async function replacePhotoFaces(photoId, faces) {
const rows = faces.map((face) => (
faceToDatabaseRow(photoId, face)
));
await db.transaction(async (trx) => {
await trx('faces')
.where({ photo_id: photoId })
.del();
if (rows.length > 0) {
await trx('faces').insert(rows);
}
});
return rows.length;
}
async function scanPhotoFacesById(photoId) {
const cleanPhotoId = String(photoId || '').trim();
if (!cleanPhotoId) {
throw new Error('photoId mancante');
}
const photo = await db('photos')
.where({ id: cleanPhotoId })
.whereNull('deleted_at')
.first();
if (!photo) {
throw new Error(
`Foto non trovata o eliminata: ${cleanPhotoId}`,
);
}
if (
photo.mime_type
&& !String(photo.mime_type).startsWith('image/')
) {
throw new Error(
`Il record non è un'immagine: `
+ `${photo.mime_type}`,
);
}
const imagePath =
await resolveFilePathForEntryId(cleanPhotoId);
if (!imagePath) {
throw new Error(
`Percorso fisico non risolto per photo_id=${cleanPhotoId}`,
);
}
let fileStat;
try {
fileStat = await fs.promises.stat(imagePath);
} catch (error) {
throw new Error(
`File non accessibile: ${imagePath}: ${error.message}`,
);
}
if (!fileStat.isFile()) {
throw new Error(
`Il percorso non è un file: ${imagePath}`,
);
}
const result = await runFaceScan(imagePath);
const insertedFaces = await replacePhotoFaces(
cleanPhotoId,
result.faces,
);
return {
photo_id: cleanPhotoId,
image_path: imagePath,
image_width: result.image.width,
image_height: result.image.height,
photo_rotation: photo.rotation ?? null,
faces_detected: result.faces.length,
faces_inserted: insertedFaces,
execution_ms: result.execution_ms,
};
}
module.exports = {
EMBEDDING_DIM,
EMBEDDING_BYTES,
embeddingToBlob,
validateFaceScanResult,
runFaceScan,
replacePhotoFaces,
scanPhotoFacesById,
};

View file

@ -0,0 +1,486 @@
const fs = require('fs');
const path = require('path');
const { execFile } = require('child_process');
const { promisify } = require('util');
const db = require('../../db/knex');
const {
resolveFilePathForEntryId,
} = require('../../utils/resolveFilePath');
const execFileAsync = promisify(execFile);
const EMBEDDING_DIM = 128;
const EMBEDDING_BYTES = EMBEDDING_DIM * 4;
const DEFAULT_FACE_CLUSTERING_ROOT =
'/home/nvme/dockerdata/prove/face-clustering-rk3588';
function getFaceClusteringRoot() {
return path.resolve(
process.env.FACE_CLUSTERING_ROOT
|| DEFAULT_FACE_CLUSTERING_ROOT,
);
}
function getFaceScanBinary() {
if (process.env.FACE_SCAN_BINARY) {
return path.resolve(process.env.FACE_SCAN_BINARY);
}
return path.join(
getFaceClusteringRoot(),
'bin',
'face_scan',
);
}
function isFiniteNumber(value) {
return (
typeof value === 'number'
&& Number.isFinite(value)
);
}
function validateImage(image) {
if (!image || typeof image !== 'object') {
throw new Error(
'Risultato face_scan non valido: image mancante',
);
}
if (
!Number.isInteger(image.width)
|| image.width <= 0
) {
throw new Error(
`Larghezza immagine non valida: ${image.width}`,
);
}
if (
!Number.isInteger(image.height)
|| image.height <= 0
) {
throw new Error(
`Altezza immagine non valida: ${image.height}`,
);
}
if (typeof image.path !== 'string') {
throw new Error(
'Percorso immagine non valido nel risultato face_scan',
);
}
}
function validateBoundingBox(bbox, faceIndex) {
if (!bbox || typeof bbox !== 'object') {
throw new Error(
`Bounding box mancante per face_index=${faceIndex}`,
);
}
for (const key of ['x', 'y', 'width', 'height']) {
if (!isFiniteNumber(bbox[key])) {
throw new Error(
`bbox.${key} non valido per face_index=${faceIndex}`,
);
}
}
if (bbox.width < 0 || bbox.height < 0) {
throw new Error(
`Bounding box negativa per face_index=${faceIndex}`,
);
}
}
function validateLandmarks(landmarks, faceIndex) {
if (!Array.isArray(landmarks)) {
throw new Error(
`Landmark non validi per face_index=${faceIndex}`,
);
}
if (landmarks.length !== 5) {
throw new Error(
`Landmark non validi per face_index=${faceIndex}: `
+ `attesi 5, ricevuti ${landmarks.length}`,
);
}
landmarks.forEach((point, landmarkIndex) => {
if (
!Array.isArray(point)
|| point.length !== 2
|| !isFiniteNumber(point[0])
|| !isFiniteNumber(point[1])
) {
throw new Error(
`Landmark ${landmarkIndex} non valido `
+ `per face_index=${faceIndex}`,
);
}
});
}
function validateEmbedding(embedding, faceIndex) {
if (!Array.isArray(embedding)) {
throw new Error(
`Embedding non valido per face_index=${faceIndex}`,
);
}
if (embedding.length !== EMBEDDING_DIM) {
throw new Error(
`Embedding non valido per face_index=${faceIndex}: `
+ `attesi ${EMBEDDING_DIM}, `
+ `ricevuti ${embedding.length}`,
);
}
for (
let embeddingIndex = 0;
embeddingIndex < embedding.length;
embeddingIndex += 1
) {
if (!isFiniteNumber(embedding[embeddingIndex])) {
throw new Error(
`Embedding non finito per face_index=${faceIndex}, `
+ `indice=${embeddingIndex}`,
);
}
}
const normSquared = embedding.reduce(
(sum, value) => sum + (value * value),
0,
);
const norm = Math.sqrt(normSquared);
if (
!Number.isFinite(norm)
|| Math.abs(norm - 1) > 0.001
) {
throw new Error(
`Embedding non normalizzato per face_index=${faceIndex}: `
+ `norm=${norm}`,
);
}
}
function validateFace(face, expectedIndex) {
if (!face || typeof face !== 'object') {
throw new Error(
`Volto non valido all'indice ${expectedIndex}`,
);
}
if (
!Number.isInteger(face.face_index)
|| face.face_index !== expectedIndex
) {
throw new Error(
`face_index non valido: atteso ${expectedIndex}, `
+ `ricevuto ${face.face_index}`,
);
}
if (!isFiniteNumber(face.score)) {
throw new Error(
`Score non valido per face_index=${expectedIndex}`,
);
}
validateBoundingBox(face.bbox, expectedIndex);
validateLandmarks(face.landmarks, expectedIndex);
validateEmbedding(face.embedding, expectedIndex);
}
function validateFaceScanResult(result) {
if (!result || typeof result !== 'object') {
throw new Error(
'Risultato face_scan non valido',
);
}
validateImage(result.image);
if (!Array.isArray(result.faces)) {
throw new Error(
'Risultato face_scan non valido: faces non è un array',
);
}
result.faces.forEach((face, faceIndex) => {
validateFace(face, faceIndex);
});
return result;
}
function embeddingToBlob(embedding) {
if (
!Array.isArray(embedding)
|| embedding.length !== EMBEDDING_DIM
) {
throw new Error(
`Embedding non valido: attesi ${EMBEDDING_DIM} valori`,
);
}
const blob = Buffer.allocUnsafe(EMBEDDING_BYTES);
embedding.forEach((value, index) => {
if (!isFiniteNumber(value)) {
throw new Error(
`Valore embedding non valido all'indice ${index}`,
);
}
blob.writeFloatLE(value, index * 4);
});
if (blob.length !== EMBEDDING_BYTES) {
throw new Error(
`Dimensione BLOB non valida: ${blob.length}`,
);
}
return blob;
}
async function verifyFaceScanBinary() {
const binaryPath = getFaceScanBinary();
try {
await fs.promises.access(
binaryPath,
fs.constants.X_OK,
);
} catch {
throw new Error(
`face_scan non trovato o non eseguibile: ${binaryPath}`,
);
}
return binaryPath;
}
async function runFaceScan(imagePath) {
const binaryPath = await verifyFaceScanBinary();
const startedAt = Date.now();
let stdout;
let stderr;
try {
const result = await execFileAsync(
binaryPath,
[imagePath],
{
encoding: 'utf8',
timeout: Number(
process.env.FACE_SCAN_TIMEOUT_MS || 300000,
),
maxBuffer: Number(
process.env.FACE_SCAN_MAX_BUFFER
|| 64 * 1024 * 1024,
),
windowsHide: true,
},
);
stdout = result.stdout;
stderr = result.stderr;
} catch (error) {
const diagnostics = [
error.message,
error.stderr,
]
.filter(Boolean)
.join('\n')
.trim();
throw new Error(
`face_scan fallito per ${imagePath}`
+ (diagnostics ? `\n${diagnostics}` : ''),
);
}
if (stderr && stderr.trim()) {
console.warn(
'[faceScanner] face_scan stderr:',
stderr.trim(),
);
}
if (!stdout || !stdout.trim()) {
throw new Error(
`face_scan non ha prodotto JSON per ${imagePath}`,
);
}
let parsed;
try {
parsed = JSON.parse(stdout);
} catch (error) {
throw new Error(
`JSON face_scan non valido per ${imagePath}: `
+ error.message,
);
}
validateFaceScanResult(parsed);
return {
...parsed,
execution_ms: Date.now() - startedAt,
};
}
function faceToDatabaseRow(photoId, face) {
const landmarks = face.landmarks;
return {
id: `face:${photoId}:${face.face_index}`,
photo_id: photoId,
face_index: face.face_index,
x: face.bbox.x,
y: face.bbox.y,
width: face.bbox.width,
height: face.bbox.height,
score: face.score,
landmark_0_x: landmarks[0][0],
landmark_0_y: landmarks[0][1],
landmark_1_x: landmarks[1][0],
landmark_1_y: landmarks[1][1],
landmark_2_x: landmarks[2][0],
landmark_2_y: landmarks[2][1],
landmark_3_x: landmarks[3][0],
landmark_3_y: landmarks[3][1],
landmark_4_x: landmarks[4][0],
landmark_4_y: landmarks[4][1],
embedding: embeddingToBlob(face.embedding),
cluster_id: null,
person_id: null,
status: 'detected',
created_at: db.fn.now(),
updated_at: db.fn.now(),
};
}
async function replacePhotoFaces(photoId, faces) {
const rows = faces.map((face) => (
faceToDatabaseRow(photoId, face)
));
await db.transaction(async (trx) => {
await trx('faces')
.where({ photo_id: photoId })
.del();
if (rows.length > 0) {
await trx('faces').insert(rows);
}
});
return rows.length;
}
async function scanPhotoFacesById(photoId) {
const cleanPhotoId = String(photoId || '').trim();
if (!cleanPhotoId) {
throw new Error('photoId mancante');
}
const photo = await db('photos')
.where({ id: cleanPhotoId })
.whereNull('deleted_at')
.first();
if (!photo) {
throw new Error(
`Foto non trovata o eliminata: ${cleanPhotoId}`,
);
}
if (
photo.mime_type
&& !String(photo.mime_type).startsWith('image/')
) {
throw new Error(
`Il record non è un'immagine: `
+ `${photo.mime_type}`,
);
}
const imagePath =
await resolveFilePathForEntryId(cleanPhotoId);
if (!imagePath) {
throw new Error(
`Percorso fisico non risolto per photo_id=${cleanPhotoId}`,
);
}
let fileStat;
try {
fileStat = await fs.promises.stat(imagePath);
} catch (error) {
throw new Error(
`File non accessibile: ${imagePath}: ${error.message}`,
);
}
if (!fileStat.isFile()) {
throw new Error(
`Il percorso non è un file: ${imagePath}`,
);
}
const result = await runFaceScan(imagePath);
const insertedFaces = await replacePhotoFaces(
cleanPhotoId,
result.faces,
);
return {
photo_id: cleanPhotoId,
image_path: imagePath,
image_width: result.image.width,
image_height: result.image.height,
photo_rotation: photo.rotation ?? null,
faces_detected: result.faces.length,
faces_inserted: insertedFaces,
execution_ms: result.execution_ms,
};
}
module.exports = {
EMBEDDING_DIM,
EMBEDDING_BYTES,
embeddingToBlob,
validateFaceScanResult,
runFaceScan,
replacePhotoFaces,
scanPhotoFacesById,
};

View file

@ -0,0 +1,108 @@
const crypto = require('crypto');
const db = require('../../db/knex');
function normalizeCoordinate(value) {
if (value == null || value === '') {
return null;
}
const coordinate = Number(value);
if (!Number.isFinite(coordinate)) {
return null;
}
return coordinate;
}
async function enqueueGeocodeJob({
entityType,
entityId,
lat,
lon,
}) {
if (
entityType !== 'photo'
&& entityType !== 'activity'
) {
throw new Error(
`Tipo entità geocode non valido: ${entityType}`
);
}
if (!entityId) {
throw new Error(
'entityId geocode mancante'
);
}
const normalizedLat = normalizeCoordinate(lat);
const normalizedLon = normalizeCoordinate(lon);
if (
normalizedLat == null
|| normalizedLon == null
) {
return {
queued: false,
reason: 'coordinates_missing',
};
}
if (
normalizedLat < -90
|| normalizedLat > 90
|| normalizedLon < -180
|| normalizedLon > 180
) {
return {
queued: false,
reason: 'coordinates_invalid',
};
}
const now = db.fn.now();
await db('geocode_jobs')
.insert({
id: crypto.randomUUID(),
entity_type: entityType,
entity_id: String(entityId),
lat: normalizedLat,
lon: normalizedLon,
status: 'pending',
attempts: 0,
last_error: null,
started_at: null,
completed_at: null,
created_at: now,
updated_at: now,
})
.onConflict([
'entity_type',
'entity_id',
])
.merge({
lat: normalizedLat,
lon: normalizedLon,
status: 'pending',
attempts: 0,
last_error: null,
started_at: null,
completed_at: null,
updated_at: now,
});
return {
queued: true,
entity_type: entityType,
entity_id: String(entityId),
lat: normalizedLat,
lon: normalizedLon,
};
}
module.exports = {
enqueueGeocodeJob,
};

View file

@ -0,0 +1,551 @@
const db = require('../../db/knex');
const loc = require('../geo');
const getElevation = require('./elevation');
function serializeError(error) {
const value =
error?.stack
|| error?.message
|| String(error);
return String(value).slice(0, 16000);
}
function buildCacheKey(lat, lon) {
return (
`${Number(lat).toFixed(4)}:`
+ `${Number(lon).toFixed(4)}`
);
}
function parseLocation(value) {
if (!value) {
return null;
}
if (typeof value === 'object') {
return value;
}
try {
return JSON.parse(value);
} catch {
return null;
}
}
function serializeLocation(value) {
if (!value) {
return null;
}
if (typeof value === 'string') {
return value;
}
return JSON.stringify(value);
}
async function claimNextPendingJob() {
return db.transaction(async (trx) => {
const job = await trx('geocode_jobs')
.where({ status: 'pending' })
.orderBy('created_at', 'asc')
.orderBy('id', 'asc')
.first();
if (!job) {
return null;
}
const updated = await trx('geocode_jobs')
.where({
id: job.id,
status: 'pending',
})
.update({
status: 'processing',
attempts: trx.raw('attempts + 1'),
last_error: null,
started_at: trx.fn.now(),
completed_at: null,
updated_at: trx.fn.now(),
});
if (updated !== 1) {
return null;
}
return trx('geocode_jobs')
.where({ id: job.id })
.first();
});
}
async function completeJob(jobId) {
const updated = await db('geocode_jobs')
.where({
id: jobId,
status: 'processing',
})
.update({
status: 'done',
last_error: null,
completed_at: db.fn.now(),
updated_at: db.fn.now(),
});
if (updated !== 1) {
throw new Error(
`Impossibile completare il job id=${jobId}`,
);
}
return 'done';
}
async function failJob(jobId, error) {
const updated = await db('geocode_jobs')
.where({
id: jobId,
status: 'processing',
})
.update({
status: 'error',
last_error: serializeError(error),
completed_at: db.fn.now(),
updated_at: db.fn.now(),
});
if (updated !== 1) {
console.error(
'[geocodeWorker] impossibile registrare errore:',
jobId,
);
return 'missing';
}
return 'error';
}
async function loadEntity(job) {
if (job.entity_type === 'photo') {
return db('photos')
.select([
'id',
'lat',
'lon',
'alt',
'location',
])
.where({ id: job.entity_id })
.whereNull('deleted_at')
.first();
}
if (job.entity_type === 'activity') {
return db('activities')
.select([
'id',
'center_latitude',
'center_longitude',
'location',
])
.where({ id: job.entity_id })
.whereNull('deleted_at')
.first();
}
throw new Error(
`Tipo entità non valido: ${job.entity_type}`,
);
}
async function loadCache(cacheKey) {
return db('geocode_cache')
.where({ cache_key: cacheKey })
.first();
}
async function saveCache({
cacheKey,
lat,
lon,
alt,
location,
}) {
const existing = await loadCache(cacheKey);
const finalAlt =
alt != null
? Number(alt)
: existing?.alt ?? null;
const finalLocation =
serializeLocation(location)
|| existing?.location
|| null;
const now = db.fn.now();
await db('geocode_cache')
.insert({
cache_key: cacheKey,
lat,
lon,
alt: finalAlt,
location: finalLocation,
hits: existing?.hits ?? 0,
created_at:
existing?.created_at || now,
updated_at: now,
})
.onConflict('cache_key')
.merge({
lat,
lon,
alt: finalAlt,
location: finalLocation,
updated_at: now,
});
}
async function recordCacheHit(cacheKey) {
await db('geocode_cache')
.where({ cache_key: cacheKey })
.update({
hits: db.raw('hits + 1'),
updated_at: db.fn.now(),
});
}
async function updateEntity({
job,
alt,
location,
}) {
const locationJson =
serializeLocation(location);
if (job.entity_type === 'photo') {
const updated = await db('photos')
.where({ id: job.entity_id })
.whereNull('deleted_at')
.update({
alt: alt ?? null,
location: locationJson,
updated_at: db.fn.now(),
});
if (updated !== 1) {
throw new Error(
`Foto non trovata: ${job.entity_id}`,
);
}
return;
}
if (job.entity_type === 'activity') {
const updated = await db('activities')
.where({ id: job.entity_id })
.whereNull('deleted_at')
.update({
location: locationJson,
updated_at: db.fn.now(),
});
if (updated !== 1) {
throw new Error(
`Attività non trovata: ${job.entity_id}`,
);
}
return;
}
throw new Error(
`Tipo entità non valido: ${job.entity_type}`,
);
}
async function processClaimedJob(job) {
if (!job?.id || !job?.entity_id) {
throw new Error('Job geocode non valido');
}
const workerStartedAt = Date.now();
console.log(
'[geocodeWorker] processing',
{
job_id: job.id,
entity_type: job.entity_type,
entity_id: job.entity_id,
attempt: job.attempts,
},
);
try {
const lat = Number(job.lat);
const lon = Number(job.lon);
if (
!Number.isFinite(lat)
|| !Number.isFinite(lon)
|| lat < -90
|| lat > 90
|| lon < -180
|| lon > 180
) {
throw new Error(
`Coordinate non valide: lat=${job.lat}, lon=${job.lon}`,
);
}
const entity = await loadEntity(job);
if (!entity) {
throw new Error(
`Entità non trovata: `
+ `${job.entity_type}/${job.entity_id}`,
);
}
const cacheKey = buildCacheKey(lat, lon);
const cached = await loadCache(cacheKey);
const altitudeFromCache =
job.entity_type === 'photo'
&& entity.alt == null
&& cached?.alt != null;
const locationFromCache =
!parseLocation(entity.location)
&& Boolean(
parseLocation(cached?.location)
);
const cacheUsed =
altitudeFromCache
|| locationFromCache;
let altitude =
job.entity_type === 'photo'
? entity.alt ?? cached?.alt ?? null
: null;
let location =
parseLocation(entity.location)
|| parseLocation(cached?.location)
|| null;
if (cacheUsed) {
await recordCacheHit(cacheKey);
}
if (
job.entity_type === 'photo'
&& altitude == null
) {
altitude = await getElevation(
lat,
lon,
);
}
if (!location) {
location = await loc(
lon,
lat,
);
}
if (!location) {
throw new Error(
`Reverse geocoding senza risultato: `
+ `lat=${lat}, lon=${lon}`,
);
}
await saveCache({
cacheKey,
lat,
lon,
alt: altitude,
location,
});
await updateEntity({
job,
alt: altitude,
location,
});
const finalStatus =
await completeJob(job.id);
const summary = {
status: finalStatus,
job_id: job.id,
entity_type: job.entity_type,
entity_id: job.entity_id,
attempt: job.attempts,
cache_key: cacheKey,
cache_used: cacheUsed,
altitude_from_cache: altitudeFromCache,
location_from_cache: locationFromCache,
altitude:
job.entity_type === 'photo'
? altitude
: null,
worker_ms:
Date.now() - workerStartedAt,
};
console.log(
'[geocodeWorker] done',
summary,
);
return summary;
} catch (error) {
const finalStatus = await failJob(
job.id,
error,
);
const summary = {
status: finalStatus,
job_id: job.id,
entity_type: job.entity_type,
entity_id: job.entity_id,
attempt: job.attempts,
error: serializeError(error),
worker_ms:
Date.now() - workerStartedAt,
};
console.error(
'[geocodeWorker] error',
summary,
);
return summary;
}
}
async function processNextJob() {
const job = await claimNextPendingJob();
if (!job) {
return null;
}
return processClaimedJob(job);
}
async function runWorker(options = {}) {
const maxJobs =
options.maxJobs == null
? null
: Number(options.maxJobs);
if (
maxJobs !== null
&& (
!Number.isInteger(maxJobs)
|| maxJobs <= 0
)
) {
throw new Error(
'maxJobs deve essere un intero positivo',
);
}
const shouldStop =
typeof options.shouldStop === 'function'
? options.shouldStop
: () => false;
const results = [];
while (
!shouldStop()
&& (
maxJobs === null
|| results.length < maxJobs
)
) {
const result = await processNextJob();
if (!result) {
break;
}
results.push(result);
}
return {
jobs_processed: results.length,
jobs_succeeded: results.filter(
(result) => result.status === 'done',
).length,
jobs_failed: results.filter(
(result) => result.status === 'error',
).length,
stopped: shouldStop(),
results,
};
}
async function resetStaleProcessingJobs(
staleMinutes = 30,
) {
const minutes = Number(staleMinutes);
if (
!Number.isFinite(minutes)
|| minutes <= 0
) {
throw new Error(
'staleMinutes deve essere positivo',
);
}
const modifier =
`-${Math.floor(minutes)} minutes`;
return db('geocode_jobs')
.where({ status: 'processing' })
.where(
'started_at',
'<',
db.raw(
"datetime('now', ?)",
[modifier],
),
)
.update({
status: 'pending',
last_error:
'Job recuperato dopo arresto del worker',
started_at: null,
completed_at: null,
updated_at: db.fn.now(),
});
}
module.exports = {
serializeError,
buildCacheKey,
claimNextPendingJob,
completeJob,
failJob,
processClaimedJob,
processNextJob,
runWorker,
resetStaleProcessingJobs,
};

69
api_v1/scanner/gps.js Normal file
View file

@ -0,0 +1,69 @@
const { exec } = require('child_process');
// -----------------------------------------------------------------------------
// FOTO: GPS da ExifReader
// -----------------------------------------------------------------------------
function extractGpsFromExif(tags) {
if (!tags?.gps) return null;
const lat = tags.gps.Latitude;
const lng = tags.gps.Longitude;
const alt = tags.gps.Altitude;
if (lat == null || lng == null) return null;
return {
lat: Number(lat),
lng: Number(lng),
alt: alt != null ? Number(alt) : null
};
}
// -----------------------------------------------------------------------------
// VIDEO: GPS via exiftool (VERSIONE ORIGINALE CHE FUNZIONA)
// -----------------------------------------------------------------------------
function extractGpsWithExiftool(videoPath) {
//console.log(videoPath);
return new Promise((resolve) => {
const cmd = `exiftool -n -G1 -a -gps:all -quicktime:all -user:all "${videoPath}"`;
exec(cmd, (err, stdout) => {
if (err || !stdout) return resolve(null);
// 1) GPS Coordinates : <lat> <lng>
const userData = stdout.match(/GPS Coordinates\s*:\s*([0-9.\-]+)\s+([0-9.\-]+)/i);
if (userData) {
return resolve({
lat: Number(userData[1]),
lng: Number(userData[2]),
alt: null
});
}
// 2) GPSLatitude / GPSLongitude
const lat1 = stdout.match(/GPSLatitude\s*:\s*([0-9.\-]+)/i);
const lng1 = stdout.match(/GPSLongitude\s*:\s*([0-9.\-]+)/i);
if (lat1 && lng1) {
return resolve({
lat: Number(lat1[1]),
lng: Number(lng1[1]),
alt: null
});
}
// 3) GPSCoordinates : <lat> <lng>
const coords = stdout.match(/GPSCoordinates\s*:\s*([0-9.\-]+)\s+([0-9.\-]+)/i);
if (coords) {
return resolve({
lat: Number(coords[1]),
lng: Number(coords[2]),
alt: null
});
}
resolve(null);
});
});
}
module.exports = { extractGpsFromExif, extractGpsWithExiftool };

43
api_v1/scanner/logger.js Normal file
View file

@ -0,0 +1,43 @@
// scanner/logger.js
const fs = require('fs');
const path = require('path');
const LOG_MODE = process.env.LOG_MODE || "console"; // console | file | both
const LOG_DIR = process.env.LOG_DIR || null;
const LOG_FILE = process.env.LOG_FILE || "scan.log";
let stream = null;
function resolveLogPath() {
if (LOG_DIR) {
return path.resolve(__dirname, "..", "..", LOG_DIR, LOG_FILE);
}
return path.resolve(__dirname, "..", "..", LOG_FILE);
}
if (LOG_MODE === "file" || LOG_MODE === "both") {
const logPath = resolveLogPath();
// crea la directory se non esiste
fs.mkdirSync(path.dirname(logPath), { recursive: true });
stream = fs.createWriteStream(logPath, { flags: "a" });
}
function ts() {
return new Date().toISOString().replace("T", " ").split(".")[0];
}
function log(message) {
const line = `${ts()} ${message}\n`;
if (LOG_MODE === "console" || LOG_MODE === "both") {
process.stdout.write(line);
}
if (LOG_MODE === "file" || LOG_MODE === "both") {
stream.write(line);
}
}
module.exports = { log };

View file

@ -0,0 +1,109 @@
// api_v1/scanner/orphanCleanup.js
const fsp = require("fs/promises");
const path = require("path");
module.exports = function createCleanupFunctions(db, RETENTION_DAYS = 30) {
// 1) Recupera gli ID dal DB per una cartella specifica
async function buildIdsListForFolder(userName, cartella) {
try {
const rows = await db("photos")
.select("id")
.where({ user: userName, cartella });
return rows.map((r) => r.id);
} catch (err) {
console.error("Errore buildIdsListForFolder:", err);
return [];
}
}
// 2) Rimuove un ID dalla lista (invariati)
function removeIdFromList(idsIndex, id) {
return idsIndex.filter((x) => x !== id);
}
// 3) Cancella thumbs dal filesystem
async function deleteThumbsById(id) {
const rec = await db("photos").where({ id }).first();
if (!rec) return false;
const thumbs = [rec.thub1, rec.thub2, rec.cast].filter(Boolean);
let deleted = false;
for (const t of thumbs) {
const abs = path.resolve(__dirname, "..", "..", "public", t);
// 🔍 Log del path reale
console.log(` 📁 Tentativo cancellazione thumb: ${abs}`);
try {
await fsp.rm(abs, { force: true });
console.log(` 🔴 Thumb eliminato: ${abs}`);
deleted = true;
} catch {}
}
return deleted;
}
// 4) Soft delete + Hard delete con retention
async function deleteFromDB(id, userName) {
const now = new Date();
const nowIso = now.toISOString();
// Recupera record attuale
const rec = await db("photos").where({ id, user: userName }).first();
if (!rec) {
console.log(`⚠️ deleteFromDB: foto ${id} non trovata`);
return false;
}
// Se NON era già soft-deleted → SOFT DELETE
if (!rec.deleted_at) {
await db("photos").where({ id, user: userName }).update({
deleted_at: nowIso,
updated_at: nowIso,
});
console.log(`🟡 Soft delete → id=${id}`);
return true;
}
// Se ERA già soft-deleted → controlla retention
const deletedAt = new Date(rec.deleted_at);
const retentionMs = RETENTION_DAYS * 24 * 60 * 60 * 1000;
const shouldHardDelete = now - deletedAt > retentionMs;
if (!shouldHardDelete) {
console.log(`🟡 Soft delete già presente → id=${id} (in retention)`);
return true;
}
// 🔥 HARD DELETE
console.log(`🔴 HARD DELETE → id=${id}`);
// 1) Elimina thumbs
await deleteThumbsById(id);
// 2) Elimina record dal DB
await db("photos").where({ id, user: userName }).del();
// 3) Registra hard delete per progressive sync
await db("deleted_hard").insert({
id,
user: userName,
deleted_at: nowIso,
});
return true;
}
return {
buildIdsListForFolder,
removeIdFromList,
deleteThumbsById,
deleteFromDB,
};
};

View file

@ -0,0 +1,100 @@
// api_v1/scanner/parseActivityFile.js
const path = require('path');
// Parser specifici
const parseFit = require('./parsers/parseFit');
const parseGpx = require('./parsers/parseGpx');
const parseTcx = require('./parsers/parseTcx');
// ---------------------------------------------------------
// parseActivityFile → rileva tipo e chiama parser specifico
// ---------------------------------------------------------
async function parseActivityFile(absPath) {
const ext = path.extname(absPath).toLowerCase();
let parsed;
if (ext === '.fit') {
parsed = await parseFit(absPath);
} else if (ext === '.gpx') {
parsed = await parseGpx(absPath);
} else if (ext === '.tcx') {
parsed = await parseTcx(absPath);
} else {
throw new Error(`Formato attività non supportato: ${ext}`);
}
// ---------------------------------------------------------
// Normalizzazione campi base
// ---------------------------------------------------------
const fileName = path.basename(absPath);
const fileExtension = ext.replace('.', '');
return {
file_name: fileName,
file_extension: fileExtension,
format: parsed.format || fileExtension.toUpperCase(),
display_name: parsed.display_name || fileName,
route_name: parsed.route_name || null,
title: parsed.title || null,
description: parsed.description || null,
notes: parsed.notes || null,
sport: parsed.sport || null,
activity_type: parsed.activity_type || null,
device_manufacturer: parsed.device_manufacturer || null,
device_model: parsed.device_model || null,
source_app: parsed.source_app || null,
start_time_millis: parsed.start_time_millis ?? null,
end_time_millis: parsed.end_time_millis ?? null,
duration_millis: parsed.duration_millis ?? null,
distance_meters: parsed.distance_meters ?? null,
elevation_gain_meters: parsed.elevation_gain_meters ?? null,
elevation_loss_meters: parsed.elevation_loss_meters ?? null,
min_elevation_meters: parsed.min_elevation_meters ?? null,
max_elevation_meters: parsed.max_elevation_meters ?? null,
file_distance_meters: parsed.file_distance_meters ?? null,
file_elevation_gain_meters: parsed.file_elevation_gain_meters ?? null,
computed_distance_meters: parsed.computed_distance_meters ?? null,
computed_elevation_gain_meters: parsed.computed_elevation_gain_meters ?? null,
average_speed_mps: parsed.average_speed_mps ?? null,
max_speed_mps: parsed.max_speed_mps ?? null,
average_pace_sec_per_km: parsed.average_pace_sec_per_km ?? null,
average_heart_rate: parsed.average_heart_rate ?? null,
max_heart_rate: parsed.max_heart_rate ?? null,
average_cadence: parsed.average_cadence ?? null,
max_cadence: parsed.max_cadence ?? null,
calories: parsed.calories ?? null,
power_avg: parsed.power_avg ?? null,
power_max: parsed.power_max ?? null,
point_count: parsed.point_count ?? 0,
segment_count: parsed.segment_count ?? 0,
lap_count: parsed.lap_count ?? 0,
min_latitude: parsed.min_latitude ?? null,
max_latitude: parsed.max_latitude ?? null,
min_longitude: parsed.min_longitude ?? null,
max_longitude: parsed.max_longitude ?? null,
center_latitude: parsed.center_latitude ?? null,
center_longitude: parsed.center_longitude ?? null,
tags: parsed.tags || [],
// IMPORTANTISSIMO: preview percorso per thumb gallery
track_preview: parsed.track_preview || [],
diagnostics: parsed.diagnostics || null,
};
}
module.exports = parseActivityFile;

View file

@ -0,0 +1,408 @@
// api_v1/scanner/parsers/parseFit.js
const { execFile } = require("child_process");
function readFitFrames(absPath) {
return new Promise(
(resolve, reject) => {
execFile(
"/home/orangepi/.local/pipx/venvs/fitdecode/bin/python",
[
"api_v1/scanner/parsers/parse_fit.py",
absPath,
],
{
maxBuffer:
1024 * 1024 * 50,
},
(error, stdout) => {
if (error) {
reject(error);
return;
}
try {
resolve(
JSON.parse(stdout),
);
} catch (parseError) {
reject(
new Error(
"Errore nel parsing JSON FIT: " +
parseError.message,
),
);
}
},
);
},
);
}
// FIT position_lat / position_long sono in semicircles.
// Conversione ufficiale: degrees = semicircles * 180 / 2^31
function fitSemicirclesToDegrees(value) {
if (value == null) return null;
const n = Number(value);
if (!Number.isFinite(n)) return null;
return n * (180 / 2147483648);
}
function buildTrackPointsFromFitRecords(
records,
) {
if (!Array.isArray(records)) {
return [];
}
const points = [];
for (const record of records) {
const fields =
record?.fields || {};
const rawLat =
fields.position_lat ??
fields.latitude ??
fields.lat;
const rawLng =
fields.position_long ??
fields.position_lon ??
fields.longitude ??
fields.lng ??
fields.lon;
const lat =
fitSemicirclesToDegrees(
rawLat,
);
const lng =
fitSemicirclesToDegrees(
rawLng,
);
if (
!Number.isFinite(lat) ||
!Number.isFinite(lng)
) {
continue;
}
const point = {
lat,
lng,
};
const altitudeRaw =
fields.altitude ??
fields.enhanced_altitude;
const altitude =
altitudeRaw != null
? Number(altitudeRaw)
: null;
if (
Number.isFinite(altitude)
) {
point.alt = altitude;
}
points.push(point);
}
return points;
}
function buildTrackPreviewFromFitRecords(
records,
maxPoints = 300,
) {
const points =
buildTrackPointsFromFitRecords(
records,
);
if (
points.length <= maxPoints
) {
return points;
}
const step =
Math.ceil(
points.length / maxPoints,
);
const preview = [];
for (
let index = 0;
index < points.length;
index += step
) {
preview.push(
points[index],
);
}
const last =
points.at(-1);
const previewLast =
preview.at(-1);
if (
last &&
(
!previewLast ||
previewLast.lat !==
last.lat ||
previewLast.lng !==
last.lng
)
) {
preview.push(last);
}
return preview;
}
async function parseFit(absPath) {
const frames =
await readFitFrames(
absPath,
);
// -------------------------------
// 1. Trova i frame principali
// -------------------------------
const sessions = frames.filter(f => f.name === "session");
const laps = frames.filter(f => f.name === "lap");
const records = frames.filter(f => f.name === "record");
const fileId = frames.find(f => f.name === "file_id");
const session = sessions[0]?.fields || {};
const file = fileId?.fields || {};
// -------------------------------
// 2. Tempi
// -------------------------------
const startTime = session.start_time || session.start_time_local || null;
const endTime = session.total_timer_time && startTime
? new Date(new Date(startTime).getTime() + session.total_timer_time * 1000)
: null;
const startMillis = startTime ? new Date(startTime).getTime() : null;
const endMillis = endTime ? endTime.getTime() : null;
const durationMillis = session.total_timer_time
? session.total_timer_time * 1000
: endMillis && startMillis
? endMillis - startMillis
: null;
// -------------------------------
// 3. Distanza / Elevazione
// -------------------------------
const distance = session.total_distance || null;
const elevGain = session.total_ascent || null;
const elevLoss = session.total_descent || null;
// -------------------------------
// 4. HR / Cadenza / Velocità
// -------------------------------
const avgHr = session.avg_heart_rate || null;
const maxHr = session.max_heart_rate || null;
const avgCad = session.avg_cadence || null;
const maxCad = session.max_cadence || null;
const avgSpeed = session.avg_speed || null;
const maxSpeed = session.max_speed || null;
// -------------------------------
// 5. Bounding box GPS
// -------------------------------
let minLat = null;
let maxLat = null;
let minLon = null;
let maxLon = null;
let minElev = null;
let maxElev = null;
for (const r of records) {
const f = r.fields || {};
const lat = fitSemicirclesToDegrees(f.position_lat);
const lon = fitSemicirclesToDegrees(f.position_long);
const eleRaw =
f.altitude ??
f.enhanced_altitude;
const ele = eleRaw != null ? Number(eleRaw) : null;
if (Number.isFinite(lat) && Number.isFinite(lon)) {
minLat = minLat == null ? lat : Math.min(minLat, lat);
maxLat = maxLat == null ? lat : Math.max(maxLat, lat);
minLon = minLon == null ? lon : Math.min(minLon, lon);
maxLon = maxLon == null ? lon : Math.max(maxLon, lon);
}
if (Number.isFinite(ele)) {
minElev = minElev == null ? ele : Math.min(minElev, ele);
maxElev = maxElev == null ? ele : Math.max(maxElev, ele);
}
}
const centerLat =
minLat != null && maxLat != null ? (minLat + maxLat) / 2 : null;
const centerLon =
minLon != null && maxLon != null ? (minLon + maxLon) / 2 : null;
// -------------------------------
// 6. Track preview per gallery
// -------------------------------
const trackPreview = buildTrackPreviewFromFitRecords(records, 300);
// Log utile temporaneo. Puoi rimuoverlo dopo il test.
console.log("[parseFit] track preview", {
file: absPath,
records: records.length,
preview: trackPreview.length,
first: trackPreview[0],
minLat,
maxLat,
minLon,
maxLon,
});
// -------------------------------
// 7. Costruzione oggetto attività finale
// -------------------------------
return {
format: "FIT",
display_name: session.sport || null,
route_name: session.sport || null,
title: session.sport || null,
description: null,
notes: null,
sport: session.sport || null,
activity_type: session.sub_sport || null,
device_manufacturer: file.manufacturer || null,
device_model: file.product || null,
source_app: null,
start_time_millis: startMillis,
end_time_millis: endMillis,
duration_millis: durationMillis,
distance_meters: distance,
elevation_gain_meters: elevGain,
elevation_loss_meters: elevLoss,
min_elevation_meters: minElev,
max_elevation_meters: maxElev,
file_distance_meters: distance,
file_elevation_gain_meters: elevGain,
computed_distance_meters: distance,
computed_elevation_gain_meters: elevGain,
average_speed_mps: avgSpeed,
max_speed_mps: maxSpeed,
average_pace_sec_per_km: avgSpeed ? 1000 / avgSpeed : null,
average_heart_rate: avgHr,
max_heart_rate: maxHr,
average_cadence: avgCad,
max_cadence: maxCad,
calories: session.total_calories || null,
power_avg: session.avg_power || null,
power_max: session.max_power || null,
point_count: records.length,
segment_count: laps.length,
lap_count: laps.length,
min_latitude: minLat,
max_latitude: maxLat,
min_longitude: minLon,
max_longitude: maxLon,
center_latitude: centerLat,
center_longitude: centerLon,
// Preview ridotta del percorso per i thumb della gallery
track_preview: trackPreview,
tags: [],
diagnostics: {
source: "FIT",
sessions: sessions.length,
laps: laps.length,
records: records.length,
track_preview_points: trackPreview.length,
coordinates_converted_from_semicircles: true,
},
};
}
async function parseFitTrackPoints(
absPath,
) {
if (!absPath) {
return [];
}
const frames =
await readFitFrames(
absPath,
);
const records =
frames.filter(
(frame) =>
frame?.name ===
"record",
);
const points =
buildTrackPointsFromFitRecords(
records,
);
/*
console.log(
"[parseFitTrackPoints] traccia completa",
{
file:
absPath,
records:
records.length,
gpsPoints:
points.length,
},
);
*/
return points;
}
module.exports =
parseFit;
module.exports.parseFitTrackPoints =
parseFitTrackPoints;

View file

@ -0,0 +1,204 @@
// api_v1/scanner/parsers/parseGpx.js
const fsp = require('fs/promises');
const xml2js = require('xml2js'); // npm install xml2js
function toMillis(date) {
if (!date) return null;
const d = new Date(date);
return isNaN(d.getTime()) ? null : d.getTime();
}
function buildTrackPreviewFromGpxPoints(points, maxPoints = 300) {
if (!Array.isArray(points)) return [];
const clean = [];
for (const p of points) {
const lat = p.$?.lat != null ? Number(p.$.lat) : null;
const lng = p.$?.lon != null ? Number(p.$.lon) : null;
const alt = p.ele != null ? Number(p.ele) : null;
if (!Number.isFinite(lat) || !Number.isFinite(lng)) continue;
const point = { lat, lng };
if (Number.isFinite(alt)) {
point.alt = alt;
}
clean.push(point);
}
if (clean.length <= maxPoints) return clean;
const step = Math.ceil(clean.length / maxPoints);
const preview = [];
for (let i = 0; i < clean.length; i += step) {
preview.push(clean[i]);
}
const last = clean[clean.length - 1];
const currentLast = preview[preview.length - 1];
if (
!currentLast ||
currentLast.lat !== last.lat ||
currentLast.lng !== last.lng
) {
preview.push(last);
}
return preview;
}
async function parseGpx(absPath) {
const xml = await fsp.readFile(absPath, 'utf8');
const parser = new xml2js.Parser({ explicitArray: false });
const gpx = await parser.parseStringPromise(xml);
const root = gpx.gpx || gpx;
const trk = Array.isArray(root.trk) ? root.trk[0] : root.trk;
const name = trk?.name || null;
const segments = trk?.trkseg
? (Array.isArray(trk.trkseg) ? trk.trkseg : [trk.trkseg])
: [];
const points = [];
for (const seg of segments) {
const segPts = seg.trkpt
? (Array.isArray(seg.trkpt) ? seg.trkpt : [seg.trkpt])
: [];
points.push(...segPts);
}
let minLat = null;
let maxLat = null;
let minLon = null;
let maxLon = null;
let minElev = null;
let maxElev = null;
let startTime = null;
let endTime = null;
let distance = null;
for (const p of points) {
const lat = p.$?.lat != null ? Number(p.$.lat) : null;
const lon = p.$?.lon != null ? Number(p.$.lon) : null;
const ele = p.ele != null ? Number(p.ele) : null;
const time = p.time || null;
if (Number.isFinite(lat) && Number.isFinite(lon)) {
minLat = minLat == null ? lat : Math.min(minLat, lat);
maxLat = maxLat == null ? lat : Math.max(maxLat, lat);
minLon = minLon == null ? lon : Math.min(minLon, lon);
maxLon = maxLon == null ? lon : Math.max(maxLon, lon);
}
if (Number.isFinite(ele)) {
minElev = minElev == null ? ele : Math.min(minElev, ele);
maxElev = maxElev == null ? ele : Math.max(maxElev, ele);
}
if (time) {
if (!startTime) startTime = time;
endTime = time;
}
}
const start_time_millis = toMillis(startTime);
const end_time_millis = toMillis(endTime);
const duration_millis =
start_time_millis && end_time_millis
? end_time_millis - start_time_millis
: null;
const centerLat =
minLat != null && maxLat != null
? (minLat + maxLat) / 2
: null;
const centerLon =
minLon != null && maxLon != null
? (minLon + maxLon) / 2
: null;
const trackPreview = buildTrackPreviewFromGpxPoints(points, 300);
return {
format: 'GPX',
display_name: name,
route_name: name,
title: name,
description: root.metadata?.desc || null,
notes: null,
sport: root.metadata?.type || null,
activity_type: null,
device_manufacturer: root.metadata?.author?.name || null,
device_model: null,
source_app: root.metadata?.author?.name || null,
start_time_millis,
end_time_millis,
duration_millis,
distance_meters: distance,
elevation_gain_meters: null,
elevation_loss_meters: null,
min_elevation_meters: minElev,
max_elevation_meters: maxElev,
file_distance_meters: distance,
file_elevation_gain_meters: null,
computed_distance_meters: distance,
computed_elevation_gain_meters: null,
average_speed_mps: null,
max_speed_mps: null,
average_pace_sec_per_km: null,
average_heart_rate: null,
max_heart_rate: null,
average_cadence: null,
max_cadence: null,
calories: null,
power_avg: null,
power_max: null,
point_count: points.length,
segment_count: segments.length,
lap_count: 0,
min_latitude: minLat,
max_latitude: maxLat,
min_longitude: minLon,
max_longitude: maxLon,
center_latitude: centerLat,
center_longitude: centerLon,
// Preview ridotta del percorso per i thumb della gallery
track_preview: trackPreview,
tags: [],
diagnostics: {
source: 'GPX',
segments: segments.length,
points: points.length,
track_preview_points: trackPreview.length
}
};
}
module.exports = parseGpx;

View file

@ -0,0 +1,298 @@
// api_v1/scanner/parsers/parseTcx.js
const fsp = require('fs/promises');
const xml2js = require('xml2js');
function toMillis(date) {
if (!date) return null;
const d = new Date(date);
return isNaN(d.getTime()) ? null : d.getTime();
}
function buildTrackPreviewFromTcxPoints(points, maxPoints = 300) {
if (!Array.isArray(points)) return [];
const clean = [];
for (const p of points) {
const pos = p.Position || {};
const lat = pos.LatitudeDegrees != null
? Number(pos.LatitudeDegrees)
: null;
const lng = pos.LongitudeDegrees != null
? Number(pos.LongitudeDegrees)
: null;
const alt = p.AltitudeMeters != null
? Number(p.AltitudeMeters)
: null;
if (!Number.isFinite(lat) || !Number.isFinite(lng)) continue;
const point = { lat, lng };
if (Number.isFinite(alt)) {
point.alt = alt;
}
clean.push(point);
}
if (clean.length <= maxPoints) return clean;
const step = Math.ceil(clean.length / maxPoints);
const preview = [];
for (let i = 0; i < clean.length; i += step) {
preview.push(clean[i]);
}
const last = clean[clean.length - 1];
const currentLast = preview[preview.length - 1];
if (
!currentLast ||
currentLast.lat !== last.lat ||
currentLast.lng !== last.lng
) {
preview.push(last);
}
return preview;
}
async function parseTcx(absPath) {
const xml = await fsp.readFile(absPath, 'utf8');
const parser = new xml2js.Parser({ explicitArray: false });
const tcx = await parser.parseStringPromise(xml);
const root = tcx.TrainingCenterDatabase || tcx;
const activities = root.Activities?.Activity
? (
Array.isArray(root.Activities.Activity)
? root.Activities.Activity
: [root.Activities.Activity]
)
: [];
const activity = activities[0] || {};
const sport = activity.$?.Sport || null;
const laps = activity.Lap
? (
Array.isArray(activity.Lap)
? activity.Lap
: [activity.Lap]
)
: [];
const tracks = [];
for (const lap of laps) {
const track = lap.Track
? (
Array.isArray(lap.Track)
? lap.Track
: [lap.Track]
)
: [];
tracks.push(...track);
}
const points = [];
for (const t of tracks) {
const tp = t.Trackpoint
? (
Array.isArray(t.Trackpoint)
? t.Trackpoint
: [t.Trackpoint]
)
: [];
points.push(...tp);
}
let minLat = null;
let maxLat = null;
let minLon = null;
let maxLon = null;
let minElev = null;
let maxElev = null;
let startTime = null;
let endTime = null;
let totalDistance = 0;
let totalCalories = 0;
let avgHr = null;
let maxHr = null;
for (const lap of laps) {
if (!startTime && lap.StartTime) {
startTime = lap.StartTime;
}
if (lap.TotalTimeSeconds && lap.StartTime) {
const lapEnd = new Date(lap.StartTime);
lapEnd.setSeconds(lapEnd.getSeconds() + Number(lap.TotalTimeSeconds));
endTime = lapEnd.toISOString();
}
if (lap.DistanceMeters) {
totalDistance += Number(lap.DistanceMeters);
}
if (lap.Calories) {
totalCalories += Number(lap.Calories);
}
const lapAvgHr = lap.AverageHeartRateBpm?.Value
? Number(lap.AverageHeartRateBpm.Value)
: null;
const lapMaxHr = lap.MaximumHeartRateBpm?.Value
? Number(lap.MaximumHeartRateBpm.Value)
: null;
if (Number.isFinite(lapAvgHr)) {
avgHr = avgHr == null ? lapAvgHr : (avgHr + lapAvgHr) / 2;
}
if (Number.isFinite(lapMaxHr)) {
maxHr = maxHr == null ? lapMaxHr : Math.max(maxHr, lapMaxHr);
}
}
for (const p of points) {
const pos = p.Position || {};
const lat = pos.LatitudeDegrees != null
? Number(pos.LatitudeDegrees)
: null;
const lon = pos.LongitudeDegrees != null
? Number(pos.LongitudeDegrees)
: null;
const ele = p.AltitudeMeters != null
? Number(p.AltitudeMeters)
: null;
const time = p.Time || null;
if (Number.isFinite(lat) && Number.isFinite(lon)) {
minLat = minLat == null ? lat : Math.min(minLat, lat);
maxLat = maxLat == null ? lat : Math.max(maxLat, lat);
minLon = minLon == null ? lon : Math.min(minLon, lon);
maxLon = maxLon == null ? lon : Math.max(maxLon, lon);
}
if (Number.isFinite(ele)) {
minElev = minElev == null ? ele : Math.min(minElev, ele);
maxElev = maxElev == null ? ele : Math.max(maxElev, ele);
}
if (time) {
if (!startTime) startTime = time;
endTime = time;
}
}
const start_time_millis = toMillis(startTime);
const end_time_millis = toMillis(endTime);
const duration_millis =
start_time_millis && end_time_millis
? end_time_millis - start_time_millis
: null;
const centerLat =
minLat != null && maxLat != null
? (minLat + maxLat) / 2
: null;
const centerLon =
minLon != null && maxLon != null
? (minLon + maxLon) / 2
: null;
const trackPreview = buildTrackPreviewFromTcxPoints(points, 300);
return {
format: 'TCX',
display_name: activity.Notes || null,
route_name: activity.Notes || null,
title: activity.Notes || null,
description: null,
notes: activity.Notes || null,
sport,
activity_type: null,
device_manufacturer: root.Author?.Name || null,
device_model: root.Author?.Build?.Version?.VersionMajor || null,
source_app: root.Author?.Name || null,
start_time_millis,
end_time_millis,
duration_millis,
distance_meters: totalDistance || null,
elevation_gain_meters: null,
elevation_loss_meters: null,
min_elevation_meters: minElev,
max_elevation_meters: maxElev,
file_distance_meters: totalDistance || null,
file_elevation_gain_meters: null,
computed_distance_meters: totalDistance || null,
computed_elevation_gain_meters: null,
average_speed_mps: null,
max_speed_mps: null,
average_pace_sec_per_km: null,
average_heart_rate: avgHr,
max_heart_rate: maxHr,
average_cadence: null,
max_cadence: null,
calories: totalCalories || null,
power_avg: null,
power_max: null,
point_count: points.length,
segment_count: tracks.length,
lap_count: laps.length,
min_latitude: minLat,
max_latitude: maxLat,
min_longitude: minLon,
max_longitude: maxLon,
center_latitude: centerLat,
center_longitude: centerLon,
// Preview ridotta del percorso per i thumb della gallery
track_preview: trackPreview,
tags: [],
diagnostics: {
source: 'TCX',
laps: laps.length,
tracks: tracks.length,
points: points.length,
track_preview_points: trackPreview.length
}
};
}
module.exports = parseTcx;

View file

@ -0,0 +1,38 @@
# parse_fit.py
import fitdecode
import json
import sys
from datetime import datetime
path = sys.argv[1]
def to_json_safe(value):
# Converte datetime → stringa ISO
if isinstance(value, datetime):
return value.isoformat()
# Converte liste/tuple ricorsivamente
if isinstance(value, (list, tuple)):
return [to_json_safe(v) for v in value]
# Converte dict ricorsivamente
if isinstance(value, dict):
return {k: to_json_safe(v) for k, v in value.items()}
# Tutto il resto viene restituito così com'è
return value
frames = []
with fitdecode.FitReader(path) as fit:
for frame in fit:
if isinstance(frame, fitdecode.FitDataMessage):
msg = {
"name": frame.name,
"fields": {}
}
for field in frame.fields:
msg["fields"][field.name] = to_json_safe(field.value)
frames.append(msg)
print(json.dumps(frames))

View file

@ -0,0 +1,24 @@
// scanner/postWithAuth.js
// Versione locale: niente HTTP, niente token, solo DB
module.exports = function createPostToDB(db) {
/**
* Inserisce o aggiorna un record nel DB SQLite
* (sostituisce completamente axios.post)
*/
async function postToDB(record) {
if (!record || !record.id) {
throw new Error("Record non valido");
}
await db('photos')
.insert(record)
.onConflict('id')
.merge();
return true;
}
return postToDB;
};

View file

@ -0,0 +1,188 @@
// api_v1/scanner/processActivityFile.js
const path = require('path');
const fsp = require('fs/promises');
const { sha256 } = require('./utils');
const { log } = require('./logger');
const parseActivityFile = require('./parseActivityFile'); // 🔥 parser FIT/GPX/TCX
const { WEB_ROOT, PATH_FULL } = require('../config');
// ---------------------------------------------------------
// processActivityFile → equivalente di processFile per le attività
// ---------------------------------------------------------
async function processActivityFile(userName, cartella, fileRelPath, absPath, ext, st) {
// 🔥 PARSING FIT/GPX/TCX
const parsed = await parseActivityFile(absPath);
// parsed contiene:
// {
// file_name,
// file_extension,
// format,
// display_name,
// route_name,
// title,
// description,
// notes,
// sport,
// activity_type,
// device_manufacturer,
// device_model,
// source_app,
// start_time_millis,
// end_time_millis,
// duration_millis,
// distance_meters,
// elevation_gain_meters,
// elevation_loss_meters,
// min_elevation_meters,
// max_elevation_meters,
// file_distance_meters,
// file_elevation_gain_meters,
// computed_distance_meters,
// computed_elevation_gain_meters,
// average_speed_mps,
// max_speed_mps,
// average_pace_sec_per_km,
// average_heart_rate,
// max_heart_rate,
// average_cadence,
// max_cadence,
// calories,
// power_avg,
// power_max,
// point_count,
// segment_count,
// lap_count,
// min_latitude,
// max_latitude,
// min_longitude,
// max_longitude,
// center_latitude,
// center_longitude,
// tags,
// diagnostics
// }
// ---------------------------------------------------------
// GPS
//
// I dati altimetrici dell'attività provengono direttamente
// dal file FIT, GPX o TCX. Il reverse geocoding viene
// completato successivamente dal geocode worker.
// ---------------------------------------------------------
let gps = null;
if (
parsed.center_latitude != null
&& parsed.center_longitude != null
) {
gps = {
lat: parsed.center_latitude,
lng: parsed.center_longitude,
alt: parsed.min_elevation_meters ?? null,
};
}
const location = null;
// ---------------------------------------------------------
// PATH FULL / RELATIVI
// ---------------------------------------------------------
const relPath = fileRelPath;
const fullPath = PATH_FULL
? path.posix.join('/activities', userName, cartella, fileRelPath)
: relPath;
// ---------------------------------------------------------
// ID deterministico
// ---------------------------------------------------------
const id = sha256(`${userName}/${cartella}/${fileRelPath}`);
// ---------------------------------------------------------
// COSTRUZIONE META (identico a normalizeActivity)
// ---------------------------------------------------------
return {
id,
user: userName,
cartella,
taken_at: new Date(parsed.start_time_millis).toISOString(),
file_name: parsed.file_name,
file_extension: parsed.file_extension,
format: parsed.format,
display_name: parsed.display_name,
route_name: parsed.route_name,
title: parsed.title,
description: parsed.description,
notes: parsed.notes,
sport: parsed.sport,
activity_type: parsed.activity_type,
device_manufacturer: parsed.device_manufacturer,
device_model: parsed.device_model,
source_app: parsed.source_app,
start_time_millis: parsed.start_time_millis,
end_time_millis: parsed.end_time_millis,
duration_millis: parsed.duration_millis,
distance_meters: parsed.distance_meters,
elevation_gain_meters: parsed.elevation_gain_meters,
elevation_loss_meters: parsed.elevation_loss_meters,
min_elevation_meters: parsed.min_elevation_meters,
max_elevation_meters: parsed.max_elevation_meters,
file_distance_meters: parsed.file_distance_meters,
file_elevation_gain_meters: parsed.file_elevation_gain_meters,
computed_distance_meters: parsed.computed_distance_meters,
computed_elevation_gain_meters: parsed.computed_elevation_gain_meters,
average_speed_mps: parsed.average_speed_mps,
max_speed_mps: parsed.max_speed_mps,
average_pace_sec_per_km: parsed.average_pace_sec_per_km,
average_heart_rate: parsed.average_heart_rate,
max_heart_rate: parsed.max_heart_rate,
average_cadence: parsed.average_cadence,
max_cadence: parsed.max_cadence,
calories: parsed.calories,
power_avg: parsed.power_avg,
power_max: parsed.power_max,
point_count: parsed.point_count,
segment_count: parsed.segment_count,
lap_count: parsed.lap_count,
min_latitude: parsed.min_latitude,
max_latitude: parsed.max_latitude,
min_longitude: parsed.min_longitude,
max_longitude: parsed.max_longitude,
center_latitude: parsed.center_latitude,
center_longitude: parsed.center_longitude,
size_bytes: st.size,
last_modified_millis: st.mtimeMs,
date_added_millis: Date.now(),
date_modified_millis: Date.now(),
last_scan_millis: Date.now(),
is_favorite: false,
is_hidden: false,
rating: 0,
tags: parsed.tags || [],
track_preview: parsed.track_preview || [],
scan_status: "ok",
scan_error: null,
diagnostics: parsed.diagnostics || null,
gps,
location,
path: fullPath
};
}
module.exports = processActivityFile;

View file

@ -0,0 +1,190 @@
// api_v1/scanner/processActivityFile.js
const path = require('path');
const fsp = require('fs/promises');
const { sha256 } = require('./utils');
const { log } = require('./logger');
const parseActivityFile = require('./parseActivityFile'); // 🔥 parser FIT/GPX/TCX
const loc = require('../geo.js');
const getElevation = require('./elevation');
const { WEB_ROOT, PATH_FULL } = require('../config');
// ---------------------------------------------------------
// processActivityFile → equivalente di processFile per le attività
// ---------------------------------------------------------
async function processActivityFile(userName, cartella, fileRelPath, absPath, ext, st) {
// 🔥 PARSING FIT/GPX/TCX
const parsed = await parseActivityFile(absPath);
// parsed contiene:
// {
// file_name,
// file_extension,
// format,
// display_name,
// route_name,
// title,
// description,
// notes,
// sport,
// activity_type,
// device_manufacturer,
// device_model,
// source_app,
// start_time_millis,
// end_time_millis,
// duration_millis,
// distance_meters,
// elevation_gain_meters,
// elevation_loss_meters,
// min_elevation_meters,
// max_elevation_meters,
// file_distance_meters,
// file_elevation_gain_meters,
// computed_distance_meters,
// computed_elevation_gain_meters,
// average_speed_mps,
// max_speed_mps,
// average_pace_sec_per_km,
// average_heart_rate,
// max_heart_rate,
// average_cadence,
// max_cadence,
// calories,
// power_avg,
// power_max,
// point_count,
// segment_count,
// lap_count,
// min_latitude,
// max_latitude,
// min_longitude,
// max_longitude,
// center_latitude,
// center_longitude,
// tags,
// diagnostics
// }
// ---------------------------------------------------------
// GPS → se manca altitudine, la calcoliamo
// ---------------------------------------------------------
let gps = null;
if (parsed.center_latitude != null && parsed.center_longitude != null) {
gps = {
lat: parsed.center_latitude,
lng: parsed.center_longitude,
alt: parsed.min_elevation_meters ?? null
};
if (gps.alt == null) {
gps.alt = await getElevation(gps.lat, gps.lng);
}
}
// ---------------------------------------------------------
// LOCATION → come per le foto
// ---------------------------------------------------------
const location = gps ? await loc(gps.lng, gps.lat) : null;
// ---------------------------------------------------------
// PATH FULL / RELATIVI
// ---------------------------------------------------------
const relPath = fileRelPath;
const fullPath = PATH_FULL
? path.posix.join('/activities', userName, cartella, fileRelPath)
: relPath;
// ---------------------------------------------------------
// ID deterministico
// ---------------------------------------------------------
const id = sha256(`${userName}/${cartella}/${fileRelPath}`);
// ---------------------------------------------------------
// COSTRUZIONE META (identico a normalizeActivity)
// ---------------------------------------------------------
return {
id,
user: userName,
cartella,
taken_at: new Date(parsed.start_time_millis).toISOString(),
file_name: parsed.file_name,
file_extension: parsed.file_extension,
format: parsed.format,
display_name: parsed.display_name,
route_name: parsed.route_name,
title: parsed.title,
description: parsed.description,
notes: parsed.notes,
sport: parsed.sport,
activity_type: parsed.activity_type,
device_manufacturer: parsed.device_manufacturer,
device_model: parsed.device_model,
source_app: parsed.source_app,
start_time_millis: parsed.start_time_millis,
end_time_millis: parsed.end_time_millis,
duration_millis: parsed.duration_millis,
distance_meters: parsed.distance_meters,
elevation_gain_meters: parsed.elevation_gain_meters,
elevation_loss_meters: parsed.elevation_loss_meters,
min_elevation_meters: parsed.min_elevation_meters,
max_elevation_meters: parsed.max_elevation_meters,
file_distance_meters: parsed.file_distance_meters,
file_elevation_gain_meters: parsed.file_elevation_gain_meters,
computed_distance_meters: parsed.computed_distance_meters,
computed_elevation_gain_meters: parsed.computed_elevation_gain_meters,
average_speed_mps: parsed.average_speed_mps,
max_speed_mps: parsed.max_speed_mps,
average_pace_sec_per_km: parsed.average_pace_sec_per_km,
average_heart_rate: parsed.average_heart_rate,
max_heart_rate: parsed.max_heart_rate,
average_cadence: parsed.average_cadence,
max_cadence: parsed.max_cadence,
calories: parsed.calories,
power_avg: parsed.power_avg,
power_max: parsed.power_max,
point_count: parsed.point_count,
segment_count: parsed.segment_count,
lap_count: parsed.lap_count,
min_latitude: parsed.min_latitude,
max_latitude: parsed.max_latitude,
min_longitude: parsed.min_longitude,
max_longitude: parsed.max_longitude,
center_latitude: parsed.center_latitude,
center_longitude: parsed.center_longitude,
size_bytes: st.size,
last_modified_millis: st.mtimeMs,
date_added_millis: Date.now(),
date_modified_millis: Date.now(),
last_scan_millis: Date.now(),
is_favorite: false,
is_hidden: false,
rating: 0,
tags: parsed.tags || [],
track_preview: parsed.track_preview || [],
scan_status: "ok",
scan_error: null,
diagnostics: parsed.diagnostics || null,
gps,
location,
path: fullPath
};
}
module.exports = processActivityFile;

View file

@ -0,0 +1,207 @@
const path = require("path");
const fsp = require("fs/promises");
const ExifReader = require("exifreader");
const sharp = require("sharp");
const { sha256, inferMimeFromExt, parseExifDateUtc } = require("./utils");
const { extractGpsFromExif, extractGpsWithExiftool } = require("./gps");
const { createVideoThumbnail, createThumbnails } = require("./thumbs");
const { probeVideo } = require("./video");
const { WEB_ROOT, PATH_FULL } = require("../config");
async function processFile(userName, cartella, fileRelPath, absPath, ext, st) {
const isVideo = [".mp4", ".mov", ".m4v"].includes(ext);
const thumbBase = path.join(
WEB_ROOT,
"photos",
userName,
"thumbs",
cartella,
path.dirname(fileRelPath),
);
await fsp.mkdir(thumbBase, { recursive: true });
// --- Nome file + estensione (dal path ASSOLUTO, sempre corretto) ---
const parsed = path.parse(absPath);
const baseName = parsed.name; // IMG_0249
const extName = parsed.ext; // .JPG
const absThumbMin = path.join(thumbBase, `${baseName}_min.jpg`);
const absThumbAvg = path.join(thumbBase, `${baseName}_avg.jpg`);
const absThumbCast = path.join(thumbBase, `${baseName}_cast.jpg`);
if (isVideo) {
await createVideoThumbnail(absPath, absThumbMin, absThumbAvg);
} else {
await createThumbnails(absPath, absThumbMin, absThumbAvg, absThumbCast);
}
// --- EXIF ---
let tags = {};
try {
tags = await ExifReader.load(absPath, { expanded: true });
} catch {}
let timeRaw = tags?.exif?.DateTimeOriginal?.value?.[0] || null;
let takenAtIso = parseExifDateUtc(timeRaw);
// Fallback per i video
if (isVideo) {
const info = await probeVideo(absPath);
// Cerca la data nei punti standard
const creationFormat = info.format?.tags?.creation_time || null;
const creationStream0 = info.streams?.[0]?.tags?.creation_time || null;
const creationStream1 = info.streams?.[1]?.tags?.creation_time || null;
// Scegli la prima disponibile
const videoDate =
creationFormat || creationStream0 || creationStream1 || null;
if (videoDate) {
// NON convertire, NON usare new Date()
takenAtIso = videoDate;
timeRaw = videoDate;
} else {
// Fallback finale: mtime del file
const fallback = new Date(st.mtimeMs).toISOString();
takenAtIso = fallback;
timeRaw = fallback;
}
}
// --- GPS ---
let gps = null;
if (isVideo) {
// i video usano exiftool
gps = await extractGpsWithExiftool(absPath);
} else {
// le foto usano exifreader
gps = extractGpsFromExif(tags);
}
// L'altitudine EXIF viene mantenuta.
// Se assente, sarà recuperata dal geocode worker.
// --- DIMENSIONI & ROTAZIONE ---
let width = null,
height = null,
duration = null,
duration_ms = null,
rotation = 0;
if (isVideo) {
const info = await probeVideo(absPath);
const stream = info.streams?.find((s) => s.width && s.height);
if (stream) {
width = stream.width;
height = stream.height;
rotation = 0;
if (stream?.tags?.rotate) {
rotation = Number(stream.tags.rotate);
}
const sdl = stream?.side_data_list;
if (sdl && Array.isArray(sdl)) {
const rotEntry = sdl.find((d) => d.rotation !== undefined);
if (rotEntry) rotation = Number(rotEntry.rotation);
const matrixEntry = sdl.find(
(d) => typeof d.displaymatrix === "string",
);
if (matrixEntry) {
const match = matrixEntry.displaymatrix.match(
/rotation of ([\-0-9]+) degrees/i,
);
if (match) rotation = Number(match[1]);
}
}
rotation = ((rotation % 360) + 360) % 360;
}
duration = info.format?.duration || null;
duration_ms = duration ? Math.round(duration * 1000) : null;
} else {
try {
const meta = await sharp(absPath).metadata();
width = meta.width || null;
height = meta.height || null;
} catch {}
try {
const raw =
tags?.exif?.Orientation?.value ??
tags?.image?.Orientation?.value ??
tags?.ifd0?.Orientation?.value ??
null;
const val = Array.isArray(raw) ? raw[0] : raw;
const map = { 1: 0, 3: 180, 6: 90, 8: 270 };
rotation = map[val] ?? 0;
} catch {}
}
const mime_type = inferMimeFromExt(ext);
const id = sha256(`${userName}/${cartella}/${fileRelPath}`);
// Reverse geocoding eseguito in modo asincrono.
const location = null;
//
// --- GESTIONE PATH FULL / RELATIVI ---
//
const relPath = fileRelPath;
const relThub1 = fileRelPath.replace(/\.[^.]+$/, "_min.jpg");
const relThub2 = fileRelPath.replace(/\.[^.]+$/, "_avg.jpg");
const relCast = isVideo ? "" : fileRelPath.replace(/\.[^.]+$/, "_cast.jpg");
const fullPath = PATH_FULL
? path.posix.join("/photos", userName, cartella, fileRelPath)
: relPath;
const fullThub1 = PATH_FULL
? path.posix.join("/photos", userName, "thumbs", cartella, relThub1)
: relThub1;
const fullThub2 = PATH_FULL
? path.posix.join("/photos", userName, "thumbs", cartella, relThub2)
: relThub2;
const fullCast = isVideo
? ""
: PATH_FULL
? path.posix.join("/photos", userName, "thumbs", cartella, relCast)
: relCast;
return {
id,
user: userName,
cartella,
name: baseName + extName, // 👈 SEMPRE CORRETTO
path: fullPath,
thub1: fullThub1,
thub2: fullThub2,
cast: fullCast,
gps,
data: timeRaw,
taken_at: takenAtIso,
mime_type,
width,
height,
rotation,
size_bytes: st.size,
mtimeMs: st.mtimeMs,
duration_ms: isVideo ? duration_ms : null,
location,
};
}
module.exports = processFile;

View file

@ -0,0 +1,213 @@
const path = require("path");
const fsp = require("fs/promises");
const ExifReader = require("exifreader");
const sharp = require("sharp");
const { sha256, inferMimeFromExt, parseExifDateUtc } = require("./utils");
const { extractGpsFromExif, extractGpsWithExiftool } = require("./gps");
const { createVideoThumbnail, createThumbnails } = require("./thumbs");
const { probeVideo } = require("./video");
const loc = require("../geo.js");
const { WEB_ROOT, PATH_FULL } = require("../config");
//const { getElevation } = require('./elevation');
const getElevation = require("./elevation");
async function processFile(userName, cartella, fileRelPath, absPath, ext, st) {
const isVideo = [".mp4", ".mov", ".m4v"].includes(ext);
const thumbBase = path.join(
WEB_ROOT,
"photos",
userName,
"thumbs",
cartella,
path.dirname(fileRelPath),
);
await fsp.mkdir(thumbBase, { recursive: true });
// --- Nome file + estensione (dal path ASSOLUTO, sempre corretto) ---
const parsed = path.parse(absPath);
const baseName = parsed.name; // IMG_0249
const extName = parsed.ext; // .JPG
const absThumbMin = path.join(thumbBase, `${baseName}_min.jpg`);
const absThumbAvg = path.join(thumbBase, `${baseName}_avg.jpg`);
const absThumbCast = path.join(thumbBase, `${baseName}_cast.jpg`);
if (isVideo) {
await createVideoThumbnail(absPath, absThumbMin, absThumbAvg);
} else {
await createThumbnails(absPath, absThumbMin, absThumbAvg, absThumbCast);
}
// --- EXIF ---
let tags = {};
try {
tags = await ExifReader.load(absPath, { expanded: true });
} catch {}
let timeRaw = tags?.exif?.DateTimeOriginal?.value?.[0] || null;
let takenAtIso = parseExifDateUtc(timeRaw);
// Fallback per i video
if (isVideo) {
const info = await probeVideo(absPath);
// Cerca la data nei punti standard
const creationFormat = info.format?.tags?.creation_time || null;
const creationStream0 = info.streams?.[0]?.tags?.creation_time || null;
const creationStream1 = info.streams?.[1]?.tags?.creation_time || null;
// Scegli la prima disponibile
const videoDate =
creationFormat || creationStream0 || creationStream1 || null;
if (videoDate) {
// NON convertire, NON usare new Date()
takenAtIso = videoDate;
timeRaw = videoDate;
} else {
// Fallback finale: mtime del file
const fallback = new Date(st.mtimeMs).toISOString();
takenAtIso = fallback;
timeRaw = fallback;
}
}
// --- GPS ---
let gps = null;
if (isVideo) {
// i video usano exiftool
gps = await extractGpsWithExiftool(absPath);
} else {
// le foto usano exifreader
gps = extractGpsFromExif(tags);
}
// --- ALTITUDINE DA SERVIZIO ESTERNO SE MANCANTE ---
if (gps && gps.lat && gps.lng) {
if (gps.alt == null) {
gps.alt = await getElevation(gps.lat, gps.lng);
}
}
// --- DIMENSIONI & ROTAZIONE ---
let width = null,
height = null,
duration = null,
duration_ms = null,
rotation = 0;
if (isVideo) {
const info = await probeVideo(absPath);
const stream = info.streams?.find((s) => s.width && s.height);
if (stream) {
width = stream.width;
height = stream.height;
rotation = 0;
if (stream?.tags?.rotate) {
rotation = Number(stream.tags.rotate);
}
const sdl = stream?.side_data_list;
if (sdl && Array.isArray(sdl)) {
const rotEntry = sdl.find((d) => d.rotation !== undefined);
if (rotEntry) rotation = Number(rotEntry.rotation);
const matrixEntry = sdl.find(
(d) => typeof d.displaymatrix === "string",
);
if (matrixEntry) {
const match = matrixEntry.displaymatrix.match(
/rotation of ([\-0-9]+) degrees/i,
);
if (match) rotation = Number(match[1]);
}
}
rotation = ((rotation % 360) + 360) % 360;
}
duration = info.format?.duration || null;
duration_ms = duration ? Math.round(duration * 1000) : null;
} else {
try {
const meta = await sharp(absPath).metadata();
width = meta.width || null;
height = meta.height || null;
} catch {}
try {
const raw =
tags?.exif?.Orientation?.value ??
tags?.image?.Orientation?.value ??
tags?.ifd0?.Orientation?.value ??
null;
const val = Array.isArray(raw) ? raw[0] : raw;
const map = { 1: 0, 3: 180, 6: 90, 8: 270 };
rotation = map[val] ?? 0;
} catch {}
}
const mime_type = inferMimeFromExt(ext);
const id = sha256(`${userName}/${cartella}/${fileRelPath}`);
const location = gps ? await loc(gps.lng, gps.lat) : null;
//
// --- GESTIONE PATH FULL / RELATIVI ---
//
const relPath = fileRelPath;
const relThub1 = fileRelPath.replace(/\.[^.]+$/, "_min.jpg");
const relThub2 = fileRelPath.replace(/\.[^.]+$/, "_avg.jpg");
const relCast = isVideo ? "" : fileRelPath.replace(/\.[^.]+$/, "_cast.jpg");
const fullPath = PATH_FULL
? path.posix.join("/photos", userName, cartella, fileRelPath)
: relPath;
const fullThub1 = PATH_FULL
? path.posix.join("/photos", userName, "thumbs", cartella, relThub1)
: relThub1;
const fullThub2 = PATH_FULL
? path.posix.join("/photos", userName, "thumbs", cartella, relThub2)
: relThub2;
const fullCast = isVideo
? ""
: PATH_FULL
? path.posix.join("/photos", userName, "thumbs", cartella, relCast)
: relCast;
return {
id,
user: userName,
cartella,
name: baseName + extName, // 👈 SEMPRE CORRETTO
path: fullPath,
thub1: fullThub1,
thub2: fullThub2,
cast: fullCast,
gps,
data: timeRaw,
taken_at: takenAtIso,
mime_type,
width,
height,
rotation,
size_bytes: st.size,
mtimeMs: st.mtimeMs,
duration_ms: isVideo ? duration_ms : null,
location,
};
}
module.exports = processFile;

View file

@ -0,0 +1,97 @@
// api_v1/scanner/recordChange.js
const { log } = require('./logger');
const wss = require('../../ws-server'); // IMPORTANTE
async function recordChange(db, photo_id, user, change_type) {
const timestamp = new Date().toISOString();
// -----------------------------------------------------
// DEDUPLICA SOLO PER "removed"
// -----------------------------------------------------
if (change_type === 'removed') {
const already = await db('photo_changes')
.where({ photo_id, user, change_type: 'removed' })
.first();
if (already) {
log(`📘 [CHANGES] removed già registrato → ${photo_id}, ma invio comunque WS`);
// -----------------------------------------------------
// 🔥 INVIA WS ANCHE SE GIÀ REGISTRATO
// -----------------------------------------------------
console.log("WS → invio evento (USER) [removed già registrato]");
try {
wss.broadcastToUser(user, {
type: "removed",
id: photo_id
});
console.log("WS → OK USER (removed già registrato)");
} catch (e) {
console.error("WS → ERRORE USER (removed già registrato):", e);
}
console.log("WS → invio evento (ADMIN) [removed già registrato]");
try {
wss.broadcastToAdmins({
type: "removed",
id: photo_id,
user
});
console.log("WS → OK ADMIN (removed già registrato)");
} catch (e) {
console.error("WS → ERRORE ADMIN (removed già registrato):", e);
}
return;
}
}
// -----------------------------------------------------
// SALVA NEL DB (added o primo removed)
// -----------------------------------------------------
await db("photo_changes").insert({
photo_id,
user,
change_type,
timestamp
});
log(`📘 [CHANGES] ${change_type} → ${photo_id}`);
// -----------------------------------------------------
// LOG PRIMA DEL BROADCAST
// -----------------------------------------------------
console.log("WS → invio evento:", { user, change_type, photo_id });
// -----------------------------------------------------
// 🔥 BROADCAST A USER
// -----------------------------------------------------
console.log("WS → invio evento (USER)");
try {
wss.broadcastToUser(user, {
type: change_type,
id: photo_id
});
console.log("WS → OK USER");
} catch (e) {
console.error("WS → ERRORE USER:", e);
}
// -----------------------------------------------------
// 🔥 BROADCAST A ADMIN
// -----------------------------------------------------
console.log("WS → invio evento (ADMIN)");
try {
wss.broadcastToAdmins({
type: change_type,
id: photo_id,
user
});
console.log("WS → OK ADMIN");
} catch (e) {
console.error("WS → ERRORE ADMIN:", e);
}
}
module.exports = recordChange;

249
api_v1/scanner/scan.md Normal file
View file

@ -0,0 +1,249 @@
scanPhotosUser(userName, db)
│
├── individua:
│ photos/<user>/original/
│
├── legge le entry dentro original/
│
├── prende SOLO le directory
│
├── countFilesUser(userDir)
│ └── ricorsivamente guarda tutti i file
│ └── conta solo quelli con estensione in SUPPORTED_EXTS
│
└── per ogni cartella:
scanPhoto(cartella, userName, db, ...)
│
└── QUI viene probabilmente
deciso cosa fare col singolo file
scanPhotosUser
│
│ per ogni cartella
▼
scanPhoto()
│
▼
scanPhotoCartella()
│
└── QUI probabilmente avviene la decisione sul file
Quindi scanPhoto():
costruisce il percorso:
const absCartella = path.join(userDir, cartella);
passa la cartella a scanPhotoCartella().
scanPhotoCartella() riempie newFiles.
solo dopo, scanPhoto() manda al server i file presenti in newFiles:
scanCartella.js non decide se la foto è già stata scansionata: decide solo quali file sono candidati alla scansione.
cartella
↓
walk() ricorsivo
↓
è directory?
├─ sì → entra dentro
└─ no
↓
estensione supportata?
├─ no → SKIP
└─ sì
↓
crea ID deterministico
↓
stat() del file
↓
yield metadata
Il valore di SUPPORTED_EXTS è quindi importante. Lo trovi in:
api_v1/config.js
La cosa ancora più interessante: l'ID
Per ogni file accettato crea:
const id = sha256(`${userName}/${cartella}/${fileRelPath}`);
Quindi l'ID non dipende dal contenuto della foto.
Dipende da:
user
+
cartella
+
percorso relativo
scanPhotoUser
↓
scanPhoto
↓
scanPhotoCartella
↓
scanCartella
↓
per ogni file con estensione supportata
↓
scanPhotoSingle(...)
↓
QUI viene probabilmente deciso:
├── già presente → non fare certe operazioni
├── nuovo → processa
└── modificato → eventualmente aggiorna
Quindi non è scanPhotoCartella a decidere se processare o meno la foto.
È scanPhotoSingle.
4. Ma c'è un altro meccanismo importante: gli orfani
Dopo scanPhotoSingle trovi:
idsSet.delete(f.id);
Questo significa:
"Ho trovato fisicamente questo file durante la scansione, quindi il suo ID non è un orfano."
Alla fine:
for (const orphanId of idsSet) {
log(`🔴 Orfano: ${orphanId}`);
await deleteThumbsById(orphanId);
await deleteFromDB(orphanId, user);
}
DB all'inizio
│
├── A
├── B
└── C
↓
scansione filesystem
↓
trovo A → idsSet.delete(A)
trovo C → idsSet.delete(C)
↓
rimane B
↓
B non esiste più sul filesystem
↓
B = ORFANO
↓
cancella thumbnail + DB
Una foto viene presa in considerazione se:
si trova dentro una cartella sotto original;
è un file, non una directory;
la sua estensione è presente in SUPPORTED_EXTS.
Poi viene generato un ID:
sha256(`${userName}/${cartella}/${fileRelPath}`)
e il file viene passato a:
scanPhotoSingle(...)
La decisione è tutta qui
Quando trova un file, fa:
let prev = await db('photos')
.select('id', 'size_bytes', 'mtimeMs', 'fast_hash', 'path')
.where({ id })
.first();
Quindi cerca nel DB una foto con lo stesso ID.
L'ID, come abbiamo visto prima, è:
sha256(`${userName}/${cartella}/${fileRelPath}`)
Quindi normalmente identifica quel preciso percorso del file.
Poi calcola:
const fastHash = sha256(`${st.size}-${st.mtimeMs}`);
Quindi il contenuto non viene hashato: il "fast hash" dipende solo da:
dimensione file + data modifica
Le 3 situazioni
1. Foto nuova
Se:
if (!prev)
cioè non esiste nel DB un record con quell'ID, fa:
const meta = await processFile(...)
Questa è la parte che legge/elabora veramente la foto.
Poi la inserisce nel DB:
await db('photos').insert(...)
e soprattutto:
newFiles.push(meta);
Quindi una foto nuova viene processata e considerata newFile.
2. Foto già presente e invariata
Questa è la situazione che probabilmente ti interessava:
if (prev.fast_hash === fastHash) {
In questo caso NON chiama processFile().
Fa solamente:
await db('photos')
.where({ id })
.update({
path: f.path,
size_bytes: st.size,
mtimeMs: st.mtimeMs,
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
return null;
Quindi:
Se l'ID esiste e size + mtimeMs sono uguali, la foto viene saltata.
Non viene riletta, non vengono rigenerate thumbnail, non viene ricalcolato GPS/location, ecc.
Questo è il vero "skip".
3. Foto modificata
Se esiste prev, ma:
prev.fast_hash !== fastHash
allora:
log(`${prefix} 🟠 Modificato: ${fileName}`);
e viene nuovamente chiamato:
const meta = await processFile(...)
Quindi la foto viene riprocessata completamente.
C'è però un caso speciale: path cambiato
Prima del controllo fast_hash c'è:
if (prev && prev.path !== f.path) {
log(`${prefix} 🟢 Nuovo/Modificato (path changed): ${fileName}`);
prev = null;
}
Questo significa:
se trovo lo stesso id ma il path nel DB è diverso da quello attuale, tratto il file come nuovo.
E quindi finisce qui:
if (!prev) {
...
await processFile(...)
...
await db('photos').insert(...)
}
Ma qui vedo una possibile anomalia
Se prev esiste già nel DB e fai:
prev = null;
poi fai:
await db('photos').insert({
id: meta.id,
...
});
Se id è una chiave unica, potresti avere un errore di duplicate key invece di un vero aggiornamento.
Però per capire se questo caso è realmente possibile bisogna vedere come viene costruito f.path e come viene gestito il DB.
TROVO FILE
│
▼
estensione supportata?
│ │
NO SI
│ │
SKIP ▼
calcolo ID
│
▼
cerco ID nel DB
│
┌────────────┴────────────┐
│ │
non esiste esiste
│ │
▼ ▼
processFile() path diverso?
│ │ │
│ SI NO
│ │ │
│ ▼ ▼
│ prev=null confronto
│ fastHash
│ │ │
│ UGUALE DIVERSO
│ │ │
│ ▼ ▼
│ SKIP processFile()
│ │
└──────────────────────────────────────┘

View file

@ -0,0 +1,160 @@
// api_v1/scanner/scanActivitiesUser.js
const path = require('path');
const fsp = require('fs/promises');
const scanActivity = require('./scanActivity');
const { log } = require('./logger');
const { WEB_ROOT, SUPPORTED_ACTIVITY_EXTS } = require('../config');
const writeJsonAtomic = require('./writeJsonAtomic');
// ---------------------------------------------------------
// ETA CALCULATOR
// ---------------------------------------------------------
function computeETA(startTime, current, total) {
if (current === 0) return 'calcolo...';
const elapsed = (Date.now() - startTime) / 1000;
const rate = current / elapsed;
const remaining = (total - current) / rate;
const m = Math.floor(remaining / 60);
const s = Math.floor(remaining % 60);
return `${m}m ${s}s`;
}
// ---------------------------------------------------------
// PRIMA PASSATA: conta TUTTI i file reali ricorsivamente
// ---------------------------------------------------------
async function countActivityFilesUser(rootDir) {
let count = 0;
async function walk(dir) {
let entries = [];
try {
entries = await fsp.readdir(dir, { withFileTypes: true });
} catch {
return;
}
for (const e of entries) {
const abs = path.join(dir, e.name);
if (e.isDirectory()) {
await walk(abs);
} else {
const ext = path.extname(e.name).toLowerCase();
if (SUPPORTED_ACTIVITY_EXTS.has(ext)) {
count++;
}
}
}
}
await walk(rootDir);
return count;
}
// ---------------------------------------------------------
// SECONDA PASSATA: scansiona SOLO le cartelle vere
// ---------------------------------------------------------
async function scanActivitiesUser(userName, db) {
log(`🔵 Inizio scan ATTIVITÀ per user=${userName}`);
const activitiesRoot = path.resolve(__dirname, '..', '..', WEB_ROOT, 'activities');
const userDir = path.join(activitiesRoot, userName, 'original');
const statusPath = path.join(activitiesRoot, 'scan_status.json');
let entries = [];
try {
entries = await fsp.readdir(userDir, { withFileTypes: true });
} catch {
log(`❌ Nessuna directory attività per utente ${userName}`);
return [];
}
// Filtra SOLO cartelle vere dentro "original"
const folders = entries
.filter(e => e.isDirectory())
.map(e => e.name);
// ---------------------------------------------------------
// RIMOZIONE CARTELLE CANCELLATE DAL FILESYSTEM
// ---------------------------------------------------------
const createDeleteFolderFunctions = require('./deleteFolder');
const { deleteFolderForUser } = createDeleteFolderFunctions(db);
const dbFolders = await db('activities')
.where({ user: userName })
.distinct('cartella')
.pluck('cartella');
for (const dbFolder of dbFolders) {
if (!folders.includes(dbFolder)) {
log(`🗑️ Cartella attività rimossa dal filesystem: ${dbFolder}`);
await deleteFolderForUser(userName, dbFolder);
}
}
// PRIMA PASSATA: conta i file reali
const TOTAL_FILES = await countActivityFilesUser(userDir);
const CURRENT = { value: 0 };
const start = Date.now();
log(`📊 Attività totali da scansionare: ${TOTAL_FILES}`);
// Stato iniziale
await writeJsonAtomic(statusPath, {
current: 0,
total: TOTAL_FILES,
percent: 0,
eta: 'calcolo...'
});
const allNewActivities = [];
// SECONDA PASSATA: UNA SOLA SCANSIONE PER CARTELLA
for (const cartella of folders) {
log(`📁 Scan cartella attività: ${cartella}`);
const newActivities = await scanActivity(
cartella,
userName,
db,
CURRENT,
TOTAL_FILES,
start,
async () => {
await writeJsonAtomic(statusPath, {
current: CURRENT.value,
total: TOTAL_FILES,
percent: TOTAL_FILES > 0
? Math.round((CURRENT.value / TOTAL_FILES) * 100)
: 100,
eta: computeETA(start, CURRENT.value, TOTAL_FILES)
});
}
);
if (newActivities?.length) {
allNewActivities.push(...newActivities);
}
}
// Stato finale
await writeJsonAtomic(statusPath, {
current: TOTAL_FILES,
total: TOTAL_FILES,
percent: 100,
eta: '0m 0s'
});
log(`🟣 Scan ATTIVITÀ COMPLETATO per user=${userName}. Nuove attività: ${allNewActivities.length}`);
return allNewActivities;
}
module.exports = scanActivitiesUser;

View file

@ -0,0 +1,114 @@
// api_v1/scanner/scanActivity.js
const path = require('path');
const { log } = require('./logger');
const scanActivityCartella = require('./scanActivityCartella');
const postWithAuth = require('./postWithAuth');
const createCleanupFunctions = require('./orphanCleanup');
const writeJsonAtomic = require('./writeJsonAtomic');
const {
WEB_ROOT,
SEND_PHOTOS,
BASE_URL
} = require('../config');
// ---------------------------------------------------------
// FORMAT TIME
// ---------------------------------------------------------
function formatTime(ms) {
const sec = Math.floor(ms / 1000);
const h = String(Math.floor(sec / 3600)).padStart(2, '0');
const m = String(Math.floor((sec % 3600) / 60)).padStart(2, '0');
const s = String(sec % 60).padStart(2, '0');
return `${h}:${m}:${s}`;
}
// ---------------------------------------------------------
// UPDATE STATUS FILE
// ---------------------------------------------------------
async function updateStatusFile(CURRENT, TOTAL_FILES, start) {
const now = Date.now();
const elapsedMs = now - start;
const avg = elapsedMs / Math.max(CURRENT.value, 1);
const remainingMs = (TOTAL_FILES - CURRENT.value) * avg;
const percent = TOTAL_FILES > 0
? Number(((CURRENT.value / TOTAL_FILES) * 100).toFixed(2))
: 100;
const status = {
current: CURRENT.value,
total: TOTAL_FILES,
percent,
eta: formatTime(remainingMs),
elapsed: formatTime(elapsedMs)
};
const statusPath = path.resolve(
__dirname,
'..',
'..',
'public/activities/scan_status.json'
);
await writeJsonAtomic(statusPath, status);
}
// ---------------------------------------------------------
// SCAN UNA SOLA CARTELLA DI ATTIVITÀ
// ---------------------------------------------------------
async function scanActivity(dir, userName, db, CURRENT, TOTAL_FILES, start, onProgress) {
const newActivities = [];
const { deleteThumbsById, deleteFromDB } = createCleanupFunctions(db);
const activitiesRoot = path.resolve(__dirname, '..', '..', WEB_ROOT, 'activities');
const userDir = path.join(activitiesRoot, userName, 'original');
const cartella = dir;
const absCartella = path.join(userDir, cartella);
log(`📁 [SCAN ACTIVITY CARTELLA] ${cartella}`);
await scanActivityCartella(
db,
userName,
cartella,
absCartella,
newActivities,
updateStatusFile,
CURRENT,
TOTAL_FILES,
start,
deleteThumbsById,
deleteFromDB,
onProgress
);
// ---------------------------------------------------------
// INVIO AL SERVER REMOTO
// ---------------------------------------------------------
if (SEND_PHOTOS && BASE_URL && newActivities.length > 0) {
log(`📤 [SEND START] newActivities=${newActivities.length}`);
for (const a of newActivities) {
try {
a.user = userName;
await postWithAuth(`${BASE_URL}/activities`, a);
log(`📥 [SENT TO SERVER] ${a.file_name}`);
} catch (err) {
log(`❌ [SERVER SENT ERROR] ${a.file_name} → ${err.message}`);
}
}
} else {
log(`⚠️ [NO SEND] newActivities=${newActivities.length}`);
}
log('📌 [DEBUG] newActivities:', JSON.stringify(newActivities, null, 2));
return newActivities;
}
module.exports = scanActivity;

View file

@ -0,0 +1,62 @@
// api_v1/scanner/scanActivityCartella.js
const scanActivityFiles = require('./scanActivityFiles');
const scanActivitySingle = require('./scanActivitySingle'); // 🔥 da creare
const { log } = require('./logger');
async function scanActivityCartella(
db,
user,
cartella,
absCartella,
newActivities,
updateStatusFile,
CURRENT,
TOTAL_FILES,
start,
deleteThumbsById,
deleteFromDB,
onProgress
) {
log(`📁 [SCAN ACTIVITY CARTELLA] ${cartella}`);
// Recupera gli ID già presenti nel DB
const rows = await db('activities')
.where({ user, cartella })
.select('id');
const idsSet = new Set(rows.map(r => r.id));
// Scansiona i file della cartella
for await (const f of scanActivityFiles(user, cartella, absCartella, db)) {
// Aggiorna il contatore
CURRENT.value++;
// Aggiorna stato avanzamento (vecchio sistema)
await updateStatusFile(CURRENT, TOTAL_FILES, start);
// Aggiorna stato avanzamento (nuovo sistema)
if (onProgress) {
await onProgress();
}
// Processa il file FIT/GPX/TCX
const meta = await scanActivitySingle(db, user, cartella, f, newActivities);
log("📌 [DEBUG] meta:", JSON.stringify(meta, null, 2));
log("📌 [DEBUG] newActivities:", JSON.stringify(newActivities, null, 2));
// Anche se identico, NON è orfano
idsSet.delete(f.id);
}
// Gestione orfani
for (const orphanId of idsSet) {
log(`🔴 [ORPHAN ACTIVITY] ${orphanId}`);
await deleteThumbsById(orphanId);
await deleteFromDB(orphanId, user);
}
}
module.exports = scanActivityCartella;

View file

@ -0,0 +1,43 @@
// api_v1/scanner/scanActivityFiles.js
const fsp = require('fs/promises');
const path = require('path');
const SUPPORTED_ACTIVITY_EXTS = new Set(['.fit', '.gpx', '.tcx']);
async function* scanActivityFiles(user, cartella, absCartella, db) {
let entries = [];
try {
entries = await fsp.readdir(absCartella, { withFileTypes: true });
} catch {
return;
}
for (const e of entries) {
if (!e.isFile()) continue;
const ext = path.extname(e.name).toLowerCase();
if (!SUPPORTED_ACTIVITY_EXTS.has(ext)) continue;
const absPath = path.join(absCartella, e.name);
// 🔥 MANCAVA QUESTO
let st;
try {
st = await fsp.stat(absPath);
} catch {
continue;
}
yield {
id: `${cartella}/${e.name}`,
file_name: e.name,
relPath: e.name, // 🔥 serve per processActivityFile
absPath,
ext,
stat: st,
path: `/activities/${user}/original/${cartella}/${e.name}` // 🔥 come le foto
};
}
}
module.exports = scanActivityFiles;

View file

@ -0,0 +1,349 @@
// api_v1/scanner/scanActivitySingle.js
const processActivityFile = require('./processActivityFile'); // parser FIT/GPX/TCX
const {
enqueueGeocodeJob,
} = require('./geocodeQueue');
const { sha256 } = require('./utils');
const { log } = require('./logger');
// Se vuoi forzare il ricalcolo anche di file invariati:
// ACTIVITY_FORCE_REPARSE=1 node ...
const FORCE_REPARSE = process.env.ACTIVITY_FORCE_REPARSE === '1';
// Normalizza location come per le foto
function normCountryCode(value) {
if (!value) return undefined;
const s = String(value).trim().toUpperCase();
return s.length ? s : undefined;
}
function normalizeLocation(location) {
if (!location) return null;
const countryCode = normCountryCode(
location.country_code ||
location.countryCode ||
location.isoCountryCode
);
const countyCode =
location.county_code ||
location.countyCode ||
undefined;
return {
...location,
country: location.country || undefined,
country_code: countryCode,
county: location.county || undefined,
county_code: countyCode,
region: location.region || undefined,
city: location.city || undefined,
postcode: location.postcode || undefined,
address: location.address || undefined,
timezone: location.timezone || undefined,
time: location.time || undefined,
};
}
function serializeJson(value, fallback = null) {
if (value == null) return fallback;
try {
return JSON.stringify(value);
} catch {
return fallback;
}
}
async function scanActivitySingle(db, user, cartella, f, newActivities, prefix = '') {
// const fileName = f.name;
const fileName = f.file_name ?? f.name ?? f.relPath ?? 'unknown';
const st = f.stat;
// ---------------------------------------------------------
// ID e fastHash PRIMA del parsing pesante
// Deve essere identico a processActivityFile:
// sha256(`${userName}/${cartella}/${fileRelPath}`)
// ---------------------------------------------------------
const id = sha256(`${user}/${cartella}/${f.relPath}`);
const fastHash = sha256(`${st.size}-${st.mtimeMs}`);
// ---------------------------------------------------------
// HARD DELETED → non reimportare attività cancellate
// ---------------------------------------------------------
const hardDeleted = await db('deleted_hard_activities')
.where({ id })
.first();
if (hardDeleted) {
log(`${prefix} ♻️ Activity restored: ${fileName}`);
await db('deleted_hard_activities')
.where({ id })
.del();
}
// ---------------------------------------------------------
// Recupero precedente usando id deterministico
// ---------------------------------------------------------
let prev = await db('activities')
.select('id', 'size_bytes', 'last_modified_millis', 'fast_hash', 'path')
.where({ id })
.first();
// ---------------------------------------------------------
// PATH CAMBIATO = tratta come nuovo/modificato
// ---------------------------------------------------------
if (prev && prev.path !== f.path) {
log(`${prefix} 🟢 Nuova/Modificata activity (path changed): ${fileName}`);
prev = null;
}
// ---------------------------------------------------------
// IDENTICO → SKIP IMMEDIATO, senza parsing FIT/GPX/TCX
// ---------------------------------------------------------
if (prev && prev.fast_hash === fastHash && !FORCE_REPARSE) {
log(`${prefix} ⏭️ Activity invariata, skip parser: ${fileName}`);
// Aggiorno solo dati leggeri. Non tocco date_modified_millis,
// così non genero changes inutili nel progressive sync.
await db('activities')
.where({ id })
.update({
path: f.path,
size_bytes: st.size,
last_modified_millis: st.mtimeMs,
last_scan_millis: Date.now(),
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
return null;
}
// ---------------------------------------------------------
// Solo qui facciamo parsing completo FIT/GPX/TCX
// ---------------------------------------------------------
const meta = await processActivityFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.path = f.path;
meta.location = normalizeLocation(meta.location);
const trackPreviewJson = serializeJson(meta.track_preview || [], null);
const tagsJson = serializeJson(meta.tags || [], null);
const diagnosticsJson = serializeJson(meta.diagnostics || null, null);
const locationJson = serializeJson(meta.location || null, null);
// ---------------------------------------------------------
// NUOVA ATTIVITÀ
// ---------------------------------------------------------
if (!prev) {
log(`${prefix} 🟢 Nuova activity: ${fileName}`);
await db('activities').insert({
id: meta.id,
user,
cartella,
file_name: meta.file_name,
file_extension: meta.file_extension,
format: meta.format,
display_name: meta.display_name,
route_name: meta.route_name,
title: meta.title,
description: meta.description,
notes: meta.notes,
sport: meta.sport,
activity_type: meta.activity_type,
device_manufacturer: meta.device_manufacturer,
device_model: meta.device_model,
source_app: meta.source_app,
start_time_millis: meta.start_time_millis,
end_time_millis: meta.end_time_millis,
duration_millis: meta.duration_millis,
distance_meters: meta.distance_meters,
elevation_gain_meters: meta.elevation_gain_meters,
elevation_loss_meters: meta.elevation_loss_meters,
min_elevation_meters: meta.min_elevation_meters,
max_elevation_meters: meta.max_elevation_meters,
file_distance_meters: meta.file_distance_meters,
file_elevation_gain_meters: meta.file_elevation_gain_meters,
computed_distance_meters: meta.computed_distance_meters,
computed_elevation_gain_meters: meta.computed_elevation_gain_meters,
average_speed_mps: meta.average_speed_mps,
max_speed_mps: meta.max_speed_mps,
average_pace_sec_per_km: meta.average_pace_sec_per_km,
average_heart_rate: meta.average_heart_rate,
max_heart_rate: meta.max_heart_rate,
average_cadence: meta.average_cadence,
max_cadence: meta.max_cadence,
calories: meta.calories,
power_avg: meta.power_avg,
power_max: meta.power_max,
point_count: meta.point_count,
segment_count: meta.segment_count,
lap_count: meta.lap_count,
min_latitude: meta.min_latitude,
max_latitude: meta.max_latitude,
min_longitude: meta.min_longitude,
max_longitude: meta.max_longitude,
center_latitude: meta.center_latitude,
center_longitude: meta.center_longitude,
size_bytes: meta.size_bytes,
last_modified_millis: meta.last_modified_millis,
date_added_millis: meta.date_added_millis,
date_modified_millis: meta.date_modified_millis,
last_scan_millis: meta.last_scan_millis,
is_favorite: meta.is_favorite,
is_hidden: meta.is_hidden,
rating: meta.rating,
tags: tagsJson,
track_preview: trackPreviewJson,
scan_status: meta.scan_status,
scan_error: meta.scan_error,
path: meta.path,
diagnostics: diagnosticsJson,
location: locationJson,
fast_hash: fastHash,
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
});
await enqueueGeocodeJob({
entityType: 'activity',
entityId: meta.id,
lat: meta.gps?.lat,
lon: meta.gps?.lng,
});
newActivities.push(meta);
return meta;
}
// ---------------------------------------------------------
// MODIFICATO oppure FORCE_REPARSE
// ---------------------------------------------------------
if (FORCE_REPARSE) {
log(`${prefix} 🔁 Force reparse activity: ${fileName}`);
} else {
log(`${prefix} 🟠 Modificato activity: ${fileName}`);
}
await db('activities')
.where({ id })
.update({
file_name: meta.file_name,
file_extension: meta.file_extension,
format: meta.format,
display_name: meta.display_name,
route_name: meta.route_name,
title: meta.title,
description: meta.description,
notes: meta.notes,
sport: meta.sport,
activity_type: meta.activity_type,
device_manufacturer: meta.device_manufacturer,
device_model: meta.device_model,
source_app: meta.source_app,
start_time_millis: meta.start_time_millis,
end_time_millis: meta.end_time_millis,
duration_millis: meta.duration_millis,
distance_meters: meta.distance_meters,
elevation_gain_meters: meta.elevation_gain_meters,
elevation_loss_meters: meta.elevation_loss_meters,
min_elevation_meters: meta.min_elevation_meters,
max_elevation_meters: meta.max_elevation_meters,
file_distance_meters: meta.file_distance_meters,
file_elevation_gain_meters: meta.file_elevation_gain_meters,
computed_distance_meters: meta.computed_distance_meters,
computed_elevation_gain_meters: meta.computed_elevation_gain_meters,
average_speed_mps: meta.average_speed_mps,
max_speed_mps: meta.max_speed_mps,
average_pace_sec_per_km: meta.average_pace_sec_per_km,
average_heart_rate: meta.average_heart_rate,
max_heart_rate: meta.max_heart_rate,
average_cadence: meta.average_cadence,
max_cadence: meta.max_cadence,
calories: meta.calories,
power_avg: meta.power_avg,
power_max: meta.power_max,
point_count: meta.point_count,
segment_count: meta.segment_count,
lap_count: meta.lap_count,
min_latitude: meta.min_latitude,
max_latitude: meta.max_latitude,
min_longitude: meta.min_longitude,
max_longitude: meta.max_longitude,
center_latitude: meta.center_latitude,
center_longitude: meta.center_longitude,
size_bytes: meta.size_bytes,
last_modified_millis: meta.last_modified_millis,
date_added_millis: meta.date_added_millis,
date_modified_millis: meta.date_modified_millis,
last_scan_millis: meta.last_scan_millis,
is_favorite: meta.is_favorite,
is_hidden: meta.is_hidden,
rating: meta.rating,
tags: tagsJson,
track_preview: trackPreviewJson,
scan_status: meta.scan_status,
scan_error: meta.scan_error,
path: meta.path,
diagnostics: diagnosticsJson,
location: locationJson,
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
await enqueueGeocodeJob({
entityType: 'activity',
entityId: meta.id,
lat: meta.gps?.lat,
lon: meta.gps?.lng,
});
newActivities.push(meta);
return meta;
}
module.exports = scanActivitySingle;

View file

@ -0,0 +1,332 @@
// api_v1/scanner/scanActivitySingle.js
const processActivityFile = require('./processActivityFile'); // parser FIT/GPX/TCX
const { sha256 } = require('./utils');
const { log } = require('./logger');
// Se vuoi forzare il ricalcolo anche di file invariati:
// ACTIVITY_FORCE_REPARSE=1 node ...
const FORCE_REPARSE = process.env.ACTIVITY_FORCE_REPARSE === '1';
// Normalizza location come per le foto
function normCountryCode(value) {
if (!value) return undefined;
const s = String(value).trim().toUpperCase();
return s.length ? s : undefined;
}
function normalizeLocation(location) {
if (!location) return null;
const countryCode = normCountryCode(
location.country_code ||
location.countryCode ||
location.isoCountryCode
);
const countyCode =
location.county_code ||
location.countyCode ||
undefined;
return {
...location,
country: location.country || undefined,
country_code: countryCode,
county: location.county || undefined,
county_code: countyCode,
region: location.region || undefined,
city: location.city || undefined,
postcode: location.postcode || undefined,
address: location.address || undefined,
timezone: location.timezone || undefined,
time: location.time || undefined,
};
}
function serializeJson(value, fallback = null) {
if (value == null) return fallback;
try {
return JSON.stringify(value);
} catch {
return fallback;
}
}
async function scanActivitySingle(db, user, cartella, f, newActivities, prefix = '') {
// const fileName = f.name;
const fileName = f.file_name ?? f.name ?? f.relPath ?? 'unknown';
const st = f.stat;
// ---------------------------------------------------------
// ID e fastHash PRIMA del parsing pesante
// Deve essere identico a processActivityFile:
// sha256(`${userName}/${cartella}/${fileRelPath}`)
// ---------------------------------------------------------
const id = sha256(`${user}/${cartella}/${f.relPath}`);
const fastHash = sha256(`${st.size}-${st.mtimeMs}`);
// ---------------------------------------------------------
// HARD DELETED → non reimportare attività cancellate
// ---------------------------------------------------------
const hardDeleted = await db('deleted_hard_activities')
.where({ id })
.first();
if (hardDeleted) {
log(`${prefix} ♻️ Activity restored: ${fileName}`);
await db('deleted_hard_activities')
.where({ id })
.del();
}
// ---------------------------------------------------------
// Recupero precedente usando id deterministico
// ---------------------------------------------------------
let prev = await db('activities')
.select('id', 'size_bytes', 'last_modified_millis', 'fast_hash', 'path')
.where({ id })
.first();
// ---------------------------------------------------------
// PATH CAMBIATO = tratta come nuovo/modificato
// ---------------------------------------------------------
if (prev && prev.path !== f.path) {
log(`${prefix} 🟢 Nuova/Modificata activity (path changed): ${fileName}`);
prev = null;
}
// ---------------------------------------------------------
// IDENTICO → SKIP IMMEDIATO, senza parsing FIT/GPX/TCX
// ---------------------------------------------------------
if (prev && prev.fast_hash === fastHash && !FORCE_REPARSE) {
log(`${prefix} ⏭️ Activity invariata, skip parser: ${fileName}`);
// Aggiorno solo dati leggeri. Non tocco date_modified_millis,
// così non genero changes inutili nel progressive sync.
await db('activities')
.where({ id })
.update({
path: f.path,
size_bytes: st.size,
last_modified_millis: st.mtimeMs,
last_scan_millis: Date.now(),
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
return null;
}
// ---------------------------------------------------------
// Solo qui facciamo parsing completo FIT/GPX/TCX
// ---------------------------------------------------------
const meta = await processActivityFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.path = f.path;
meta.location = normalizeLocation(meta.location);
const trackPreviewJson = serializeJson(meta.track_preview || [], null);
const tagsJson = serializeJson(meta.tags || [], null);
const diagnosticsJson = serializeJson(meta.diagnostics || null, null);
const locationJson = serializeJson(meta.location || null, null);
// ---------------------------------------------------------
// NUOVA ATTIVITÀ
// ---------------------------------------------------------
if (!prev) {
log(`${prefix} 🟢 Nuova activity: ${fileName}`);
await db('activities').insert({
id: meta.id,
user,
cartella,
file_name: meta.file_name,
file_extension: meta.file_extension,
format: meta.format,
display_name: meta.display_name,
route_name: meta.route_name,
title: meta.title,
description: meta.description,
notes: meta.notes,
sport: meta.sport,
activity_type: meta.activity_type,
device_manufacturer: meta.device_manufacturer,
device_model: meta.device_model,
source_app: meta.source_app,
start_time_millis: meta.start_time_millis,
end_time_millis: meta.end_time_millis,
duration_millis: meta.duration_millis,
distance_meters: meta.distance_meters,
elevation_gain_meters: meta.elevation_gain_meters,
elevation_loss_meters: meta.elevation_loss_meters,
min_elevation_meters: meta.min_elevation_meters,
max_elevation_meters: meta.max_elevation_meters,
file_distance_meters: meta.file_distance_meters,
file_elevation_gain_meters: meta.file_elevation_gain_meters,
computed_distance_meters: meta.computed_distance_meters,
computed_elevation_gain_meters: meta.computed_elevation_gain_meters,
average_speed_mps: meta.average_speed_mps,
max_speed_mps: meta.max_speed_mps,
average_pace_sec_per_km: meta.average_pace_sec_per_km,
average_heart_rate: meta.average_heart_rate,
max_heart_rate: meta.max_heart_rate,
average_cadence: meta.average_cadence,
max_cadence: meta.max_cadence,
calories: meta.calories,
power_avg: meta.power_avg,
power_max: meta.power_max,
point_count: meta.point_count,
segment_count: meta.segment_count,
lap_count: meta.lap_count,
min_latitude: meta.min_latitude,
max_latitude: meta.max_latitude,
min_longitude: meta.min_longitude,
max_longitude: meta.max_longitude,
center_latitude: meta.center_latitude,
center_longitude: meta.center_longitude,
size_bytes: meta.size_bytes,
last_modified_millis: meta.last_modified_millis,
date_added_millis: meta.date_added_millis,
date_modified_millis: meta.date_modified_millis,
last_scan_millis: meta.last_scan_millis,
is_favorite: meta.is_favorite,
is_hidden: meta.is_hidden,
rating: meta.rating,
tags: tagsJson,
track_preview: trackPreviewJson,
scan_status: meta.scan_status,
scan_error: meta.scan_error,
path: meta.path,
diagnostics: diagnosticsJson,
location: locationJson,
fast_hash: fastHash,
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
});
newActivities.push(meta);
return meta;
}
// ---------------------------------------------------------
// MODIFICATO oppure FORCE_REPARSE
// ---------------------------------------------------------
if (FORCE_REPARSE) {
log(`${prefix} 🔁 Force reparse activity: ${fileName}`);
} else {
log(`${prefix} 🟠 Modificato activity: ${fileName}`);
}
await db('activities')
.where({ id })
.update({
file_name: meta.file_name,
file_extension: meta.file_extension,
format: meta.format,
display_name: meta.display_name,
route_name: meta.route_name,
title: meta.title,
description: meta.description,
notes: meta.notes,
sport: meta.sport,
activity_type: meta.activity_type,
device_manufacturer: meta.device_manufacturer,
device_model: meta.device_model,
source_app: meta.source_app,
start_time_millis: meta.start_time_millis,
end_time_millis: meta.end_time_millis,
duration_millis: meta.duration_millis,
distance_meters: meta.distance_meters,
elevation_gain_meters: meta.elevation_gain_meters,
elevation_loss_meters: meta.elevation_loss_meters,
min_elevation_meters: meta.min_elevation_meters,
max_elevation_meters: meta.max_elevation_meters,
file_distance_meters: meta.file_distance_meters,
file_elevation_gain_meters: meta.file_elevation_gain_meters,
computed_distance_meters: meta.computed_distance_meters,
computed_elevation_gain_meters: meta.computed_elevation_gain_meters,
average_speed_mps: meta.average_speed_mps,
max_speed_mps: meta.max_speed_mps,
average_pace_sec_per_km: meta.average_pace_sec_per_km,
average_heart_rate: meta.average_heart_rate,
max_heart_rate: meta.max_heart_rate,
average_cadence: meta.average_cadence,
max_cadence: meta.max_cadence,
calories: meta.calories,
power_avg: meta.power_avg,
power_max: meta.power_max,
point_count: meta.point_count,
segment_count: meta.segment_count,
lap_count: meta.lap_count,
min_latitude: meta.min_latitude,
max_latitude: meta.max_latitude,
min_longitude: meta.min_longitude,
max_longitude: meta.max_longitude,
center_latitude: meta.center_latitude,
center_longitude: meta.center_longitude,
size_bytes: meta.size_bytes,
last_modified_millis: meta.last_modified_millis,
date_added_millis: meta.date_added_millis,
date_modified_millis: meta.date_modified_millis,
last_scan_millis: meta.last_scan_millis,
is_favorite: meta.is_favorite,
is_hidden: meta.is_hidden,
rating: meta.rating,
tags: tagsJson,
track_preview: trackPreviewJson,
scan_status: meta.scan_status,
scan_error: meta.scan_error,
path: meta.path,
diagnostics: diagnosticsJson,
location: locationJson,
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
newActivities.push(meta);
return meta;
}
module.exports = scanActivitySingle;

211
api_v1/scanner/scanAuto.js Normal file
View file

@ -0,0 +1,211 @@
const path = require('path');
const fsp = require('fs/promises');
const scanCartella = require('./scanCartella');
const processFile = require('./processFile');
const { sha256 } = require('./utils');
const createCleanupFunctions = require('./orphanCleanup');
const { WEB_ROOT } = require('../config');
const { log } = require('./logger');
// ---------------------------------------------------------
// NORMALIZZAZIONE PATH (toglie /app/...)
// ---------------------------------------------------------
function normalizeRelPath(user, dirPath, fileName) {
if (!dirPath) return "";
let rel = dirPath.replace(/\\/g, "/");
// Rimuove tutto fino a /photos/
rel = rel.replace(/^.*\/photos\//, "");
// Ora rel = "Fabio/original/2017Irlanda/.../"
const prefix = `${user}/original/`;
if (rel.startsWith(prefix)) {
rel = rel.slice(prefix.length);
}
if (rel.startsWith("/")) rel = rel.slice(1);
// Aggiunge il file
if (fileName) {
rel = `${rel}/${fileName}`.replace(/\/+/g, "/");
}
return rel; // es: "2017Irlanda19-29ago/IMG_0120.JPG"
}
// ---------------------------------------------------------
// CALCOLO ID IDENTICO AL VECCHIO SCAN
// ---------------------------------------------------------
function computeId(user, relPath) {
const parts = relPath.split('/').filter(Boolean);
const cartella = parts[0];
const fileRelPath = parts.slice(1).join('/');
const id = sha256(`${user}/${cartella}/${fileRelPath}`);
return { id, cartella, fileRelPath };
}
// ---------------------------------------------------------
// 1) ADD FILE
// ---------------------------------------------------------
async function handleAddFile(user, dirPath, fileName, db) {
const relPath = normalizeRelPath(user, dirPath, fileName);
log(`🟦 [ADD] user=${user} file=${relPath}`);
// 🔥 Ricostruzione identica al vecchio scanPhoto
const parts = relPath.split('/');
const cartella = parts[0]; // es: "2017Irlanda19-29ago"
const fileRelPath = parts.slice(1).join('/'); // es: "IMG_0116.JPG"
const photosRoot = path.resolve(__dirname, '..', '..', WEB_ROOT, 'photos');
const absPath = path.join(photosRoot, user, 'original', cartella, fileRelPath);
const ext = path.extname(fileName).toLowerCase();
const st = await fsp.stat(absPath);
// 🔥 FIX: processFile deve ricevere fileRelPath completo di "original/"
const meta = await processFile(
user,
cartella,
path.posix.join('original', cartella, fileRelPath),
absPath,
ext,
st
);
// Inserimento identico al vecchio scanPhoto
const row = {
id: meta.id,
user,
cartella,
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
cast: meta.cast,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms ?? null,
taken_at: meta.taken_at ?? null,
data: meta.data ?? null,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
location: meta.location ? JSON.stringify(meta.location) : null
};
await db('photos')
.insert(row)
.onConflict('id')
.merge();
log(`🟢 [ADD] Inserito/aggiornato id=${meta.id}`);
return row;
}
// ---------------------------------------------------------
// 2) DELETE FILE
// ---------------------------------------------------------
async function handleDeleteFile(user, dirPath, fileName, db) {
const relPath = normalizeRelPath(user, dirPath, fileName);
log(`🟥 [DEL] user=${user} file=${relPath}`);
const { id } = computeId(user, relPath);
const { deleteThumbsById, deleteFromDB } = createCleanupFunctions(db);
await deleteThumbsById(id);
await deleteFromDB(id, user);
log(`🔴 [DEL] File eliminato id=${id}`);
return { deleted: id };
}
// ---------------------------------------------------------
// 3) ADD DIRECTORY
// ---------------------------------------------------------
async function handleAddDir(user, dir, db) {
log(`🟦 [ADD_DIR] user=${user} dir=${dir}`);
const photosRoot = path.resolve(__dirname, '..', '..', WEB_ROOT, 'photos');
const absDir = path.join(photosRoot, user, 'original', dir);
log(`📁 [ADD_DIR] absDir=${absDir}`);
let results = [];
for await (const f of scanCartella(user, dir, absDir, db)) {
log(`📄 [ADD_DIR] File trovato: ${f.relPath}`);
const meta = await processFile(
user,
dir,
f.relPath,
f.absPath,
f.ext,
f.stat
);
await db('photos').insert(meta).onConflict('id').merge();
log(`🟢 [ADD_DIR] Inserito/aggiornato id=${meta.id} name=${meta.name}`);
results.push(meta);
}
log(`🟣 [ADD_DIR] Completato. Files=${results.length}`);
return results;
}
// ---------------------------------------------------------
// 4) DELETE DIRECTORY
// ---------------------------------------------------------
async function handleDeleteDir(user, dir, db) {
log(`🟥 [DEL_DIR] user=${user} dir=${dir}`);
const rows = await db('photos')
.where({ user, cartella: dir })
.select('id');
const { deleteThumbsById, deleteFromDB } = createCleanupFunctions(db);
log(`📁 [DEL_DIR] Trovati ${rows.length} file da eliminare`);
for (const r of rows) {
log(`🗑️ [DEL_DIR] Eliminazione id=${r.id}`);
await deleteThumbsById(r.id);
await deleteFromDB(r.id, user);
}
log(`🔴 [DEL_DIR] Cartella eliminata. Totale file=${rows.length}`);
return { deleted: rows.length };
}
module.exports = {
handleAddFile,
handleDeleteFile,
handleAddDir,
handleDeleteDir
};

View file

@ -0,0 +1,68 @@
// api_v1/scanner/scanCartella.js
const path = require('path');
const fsp = require('fs/promises');
const fs = require('fs');
const { sha256 } = require('./utils');
const { SUPPORTED_EXTS } = require('../config');
const { log } = require('./logger');
/**
* Scansiona ricorsivamente la cartella:
* /photos/<user>/original/<cartella>/
*
* Restituisce (yield) i metadati dei file trovati.
*/
async function* scanCartella(userName, cartella, absCartella, db) {
async function* walk(currentAbs, relPath = '') {
let entries = [];
try {
entries = await fsp.readdir(currentAbs, { withFileTypes: true });
} catch {
return;
}
console.log(entries);
for (const e of entries) {
const absPath = path.join(currentAbs, e.name);
if (e.isDirectory()) {
yield* walk(absPath, path.join(relPath, e.name));
continue;
}
const ext = path.extname(e.name).toLowerCase();
if (!SUPPORTED_EXTS.has(ext)) continue;
const fileRelPath = relPath ? `${relPath}/${e.name}` : e.name;
// ID deterministico basato sul percorso
const id = sha256(`${userName}/${cartella}/${fileRelPath}`);
let st;
try {
st = await fsp.stat(absPath);
} catch {
continue;
}
log(`📂 scanCartella → user=${userName} cartella=${cartella} relPath=${fileRelPath} id=${id}`);
yield {
id,
user: userName,
cartella,
name: e.name,
relPath: fileRelPath,
absPath,
ext,
stat: st,
path: `/photos/${userName}/original/${cartella}/${fileRelPath}`
};
}
}
yield* walk(absCartella);
}
module.exports = scanCartella;

235
api_v1/scanner/scanFile.js Normal file
View file

@ -0,0 +1,235 @@
// api_v1/scanner/scanFile.js
const path = require('path');
const fsp = require('fs/promises');
const { sha256 } = require('./utils');
const { SUPPORTED_EXTS } = require('../config');
const { log } = require('./logger');
/**
* scanFile: genera informazioni su UN singolo file
* Restituisce lo stesso formato di scanCartella, ma senza recursion.
*
* Parametri:
* - userName
* - cartella
* - absFile (percorso assoluto del file)
* - db (opzionale)
*/
async function scanFileEntry(userName, cartella, absFile, db) {
const ext = path.extname(absFile).toLowerCase();
if (!SUPPORTED_EXTS.has(ext)) {
log(`⛔ Estensione non supportata: ${ext}`);
return null;
}
const name = path.basename(absFile);
const relPath = name; // singolo file → niente struttura ricorsiva
let st;
try {
st = await fsp.stat(absFile);
} catch {
log(`⛔ Impossibile leggere stat per ${absFile}`);
return null;
}
const id = sha256(`${userName}/${cartella}/${relPath}`);
log(`📄 scanFile → user=${userName} cartella=${cartella} file=${name} id=${id}`);
return {
id,
user: userName,
cartella,
name,
relPath,
absPath: absFile,
ext,
stat: st,
path: `/photos/${userName}/original/${cartella}/${relPath}`
};
}
async function scanFile(userName, cartella, absFile, db) {
const f = await scanFile(user, cart, absFile);
const fileName = f.name;
const id = f.id;
const st = f.stat;
let prev = await db("photos")
.select("id", "size_bytes", "mtimeMs", "_indexHash", "fast_hash", "path")
.where({ id })
.first();
if (prev && LOG_VERBOSE) {
log(`${prefix} ⚪ Invariato: ${fileName}`);
}
const fastHash = sha256(`${st.size}-${st.mtimeMs}`);
// ---------------------------------------------------------
// PATH CAMBIATO = NUOVO FILE
// ---------------------------------------------------------
if (prev && prev.path !== f.path) {
log(`${prefix} 🟢 Nuovo/Modificato: ${fileName}`);
prev = null;
}
// ---------------------------------------------------------
// NUOVO FILE
// ---------------------------------------------------------
if (!prev) {
log(`${prefix} 🟢 Nuovo/Modificato: ${fileName}`);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
// INSERT CORRETTO (senza colonna gps)
await db("photos").insert({
id: meta.id,
user,
cartella,
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
cast: meta.cast,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
location: meta.location ? JSON.stringify(meta.location) : null,
_indexHash: meta._indexHash,
fast_hash: fastHash
});
await db('photo_changes').insert({
photo_id: meta.id,
user,
change_type: 'added',
timestamp: new Date().toISOString()
});
newFiles.push(meta);
//log(`🟢 [PUSH newFiles] id=${meta.id} path=${meta.path}`);
return;
}
// ---------------------------------------------------------
// FAST-SIZE-SKIP
// ---------------------------------------------------------
if (prev.size_bytes === st.size) {
//log(`🔵 [FAST-SIZE-SKIP] id=${id}`);
await db("photos")
.where({ id })
.update({
path: f.path,
size_bytes: st.size,
mtimeMs: st.mtimeMs,
fast_hash: fastHash
});
continue;
}
// ---------------------------------------------------------
// FAST-HASH-SKIP
// ---------------------------------------------------------
if (prev.fast_hash === fastHash) {
//log(`🔵 [FAST-HASH-SKIP] id=${id}`);
await db("photos")
.where({ id })
.update({
path: f.path,
size_bytes: st.size,
mtimeMs: st.mtimeMs
});
continue;
}
// ---------------------------------------------------------
// MODIFICATO
// ---------------------------------------------------------
//log(`🟠 [FULL-SCAN] id=${id}`);
log(`${prefix} 🟠 Nuovo/Modificato: ${fileName}`);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
await db("photos")
.insert({
id: meta.id,
user,
cartella,
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
cast: meta.cast,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
location: meta.location ? JSON.stringify(meta.location) : null,
_indexHash: meta._indexHash,
fast_hash: fastHash
})
.onConflict("id")
.merge();
await db('photo_changes').insert({
photo_id: meta.id,
user,
change_type: 'updated',
timestamp: new Date().toISOString()
});
newFiles.push(meta);
log(`${prefix} 🟠 Nuovo/Modificato al server ${fileName}`);
}
}
}
module.exports = scanFile;

View file

@ -0,0 +1,53 @@
// api_v1/scanner/scanFile.js
const path = require('path');
const fsp = require('fs/promises');
const { sha256 } = require('./utils');
const { SUPPORTED_EXTS } = require('../config');
const { log } = require('./logger');
/**
* scanFile: genera informazioni su UN singolo file
* Restituisce lo stesso formato di scanCartella, ma senza recursion.
*
* Parametri:
* - userName
* - cartella
* - absFile (percorso assoluto del file)
* - db (opzionale)
*/
async function scanFile(userName, cartella, absFile, db) {
const ext = path.extname(absFile).toLowerCase();
if (!SUPPORTED_EXTS.has(ext)) {
log(`⛔ Estensione non supportata: ${ext}`);
return null;
}
const name = path.basename(absFile);
const relPath = name; // singolo file → niente struttura ricorsiva
let st;
try {
st = await fsp.stat(absFile);
} catch {
log(`⛔ Impossibile leggere stat per ${absFile}`);
return null;
}
const id = sha256(`${userName}/${cartella}/${relPath}`);
log(`📄 scanFile → user=${userName} cartella=${cartella} file=${name} id=${id}`);
return {
id,
user: userName,
cartella,
name,
relPath,
absPath: absFile,
ext,
stat: st,
path: `/photos/${userName}/original/${cartella}/${relPath}`
};
}
module.exports = scanFile;

View file

@ -0,0 +1,55 @@
// api_v1/scanner/scanFileEntry.js
const path = require('path');
const fsp = require('fs/promises');
const { sha256 } = require('./utils');
const { SUPPORTED_EXTS } = require('../config');
const { log } = require('./logger');
/**
* scanFile: genera informazioni su UN singolo file
* Restituisce lo stesso formato di scanCartella, ma senza recursion.
*
* Parametri:
* - userName
* - cartella
* - absFile (percorso assoluto del file)
* - db (opzionale)
*/
async function scanFile(userName, cartella, absFile, db) {
const ext = path.extname(absFile).toLowerCase();
if (!SUPPORTED_EXTS.has(ext)) {
log(`⛔ Estensione non supportata: ${ext}`);
return null;
}
const name = path.basename(absFile);
const relPath = name; // singolo file → niente struttura ricorsiva
// ID deterministico (funziona anche se il file non esiste)
const id = sha256(`${userName}/${cartella}/${relPath}`);
let st = null;
try {
st = await fsp.stat(absFile);
} catch {
// File NON esiste → caso DEL
log(`⛔ Impossibile leggere stat per ${absFile} (file mancante)`);
// Continuiamo comunque: l'ID è valido e serve per DEL
}
log(`📄 scanFile → user=${userName} cartella=${cartella} file=${name} id=${id}`);
return {
id,
user: userName,
cartella,
name,
relPath,
absPath: absFile,
ext,
stat: st, // può essere null
path: `/photos/${userName}/original/${cartella}/${relPath}`
};
}
module.exports = scanFile;

View file

@ -0,0 +1,76 @@
// api_v1/scanner/scanNewCartella.js
const path = require('path');
const fsp = require('fs/promises');
const { WEB_ROOT, SUPPORTED_EXTS } = require('../config');
const scanPhoto = require('./scanPhoto');
// ---------------------------------------------------------
// Conta ricorsivamente i file in /photos/<user>/original/<folder>/
// ---------------------------------------------------------
async function countFilesInUserFolder(userName, folderName) {
const rootDir = path.join(WEB_ROOT, userName, "original", folderName);
let count = 0;
async function walk(dir) {
let entries = [];
try {
entries = await fsp.readdir(dir, { withFileTypes: true });
} catch {
return;
}
for (const e of entries) {
const abs = path.join(dir, e.name);
if (e.isDirectory()) {
await walk(abs);
} else {
const ext = path.extname(e.name).toLowerCase();
if (SUPPORTED_EXTS.has(ext)) {
count++;
}
}
}
}
await walk(rootDir);
return count;
}
// ---------------------------------------------------------
// Crea thumbs/<folder> e scansiona la cartella come fa /scan
// ---------------------------------------------------------
async function scanNewCartella(userName, folderName, db) {
// 1) crea la cartella root in thumbs
const thumbsDir = path.join(WEB_ROOT, userName, "thumbs", folderName);
await fsp.mkdir(thumbsDir, { recursive: true });
// 2) conta i file ricorsivamente
const TOTAL_FILES = await countFilesInUserFolder(userName, folderName);
// 3) esegui lo scan della cartella
const CURRENT = { value: 0 };
const start = Date.now();
// ⭐ newFiles conterrà i meta delle foto nuove
const newFiles = await scanPhoto(
folderName,
userName,
db,
CURRENT,
TOTAL_FILES,
start
);
return {
ok: true,
folder: folderName,
totalFiles: TOTAL_FILES,
newFiles
};
}
module.exports = {
scanNewCartella,
countFilesInUserFolder
};

111
api_v1/scanner/scanPhoto.js Normal file
View file

@ -0,0 +1,111 @@
// api_v1/scanner/scanPhoto.js
const path = require('path');
const { log } = require('./logger');
const scanPhotoCartella = require('./scanPhotoCartella');
const postWithAuth = require('./postWithAuth');
const createCleanupFunctions = require('./orphanCleanup');
const writeJsonAtomic = require('./writeJsonAtomic');
const {
WEB_ROOT,
SEND_PHOTOS,
BASE_URL
} = require('../config');
// ---------------------------------------------------------
// FORMAT TIME
// ---------------------------------------------------------
function formatTime(ms) {
const sec = Math.floor(ms / 1000);
const h = String(Math.floor(sec / 3600)).padStart(2, '0');
const m = String(Math.floor((sec % 3600) / 60)).padStart(2, '0');
const s = String(sec % 60).padStart(2, '0');
return `${h}:${m}:${s}`;
}
// ---------------------------------------------------------
// UPDATE STATUS FILE
// ---------------------------------------------------------
async function updateStatusFile(CURRENT, TOTAL_FILES, start) {
const now = Date.now();
const elapsedMs = now - start;
const avg = elapsedMs / Math.max(CURRENT.value, 1);
const remainingMs = (TOTAL_FILES - CURRENT.value) * avg;
const percent = TOTAL_FILES > 0
? Number(((CURRENT.value / TOTAL_FILES) * 100).toFixed(2))
: 100;
const status = {
current: CURRENT.value,
total: TOTAL_FILES,
percent,
eta: formatTime(remainingMs),
elapsed: formatTime(elapsedMs)
};
const statusPath = path.resolve(
__dirname,
'..',
'..',
'public/photos/scan_status.json'
);
await writeJsonAtomic(statusPath, status);
}
// ---------------------------------------------------------
// SCAN UNA SOLA CARTELLA
// ---------------------------------------------------------
async function scanPhoto(dir, userName, db, CURRENT, TOTAL_FILES, start, onProgress) {
const newFiles = [];
const { deleteThumbsById, deleteFromDB } = createCleanupFunctions(db);
const photosRoot = path.resolve(__dirname, '..', '..', WEB_ROOT, 'photos');
const userDir = path.join(photosRoot, userName, 'original');
const cartella = dir;
const absCartella = path.join(userDir, cartella);
log(`📁 [SCAN CARTELLA] ${cartella}`);
await scanPhotoCartella(
db,
userName,
cartella,
absCartella,
newFiles,
updateStatusFile,
CURRENT,
TOTAL_FILES,
start,
deleteThumbsById,
deleteFromDB,
onProgress
);
// ---------------------------------------------------------
// INVIO AL SERVER REMOTO
// ---------------------------------------------------------
if (SEND_PHOTOS && BASE_URL && newFiles.length > 0) {
log(`📤 [SEND START] newFiles=${newFiles.length}`);
for (const p of newFiles) {
try {
p.user = userName;
await postWithAuth(`${BASE_URL}/photos`, p);
log(`📥 [SENT TO SERVER] ${p.name}`);
} catch (err) {
log(`❌ [SERVER SENT ERROR] ${p.name} → ${err.message}`);
}
}
} else {
log(`⚠️ [NO SEND] newFiles=${newFiles.length}`);
}
return newFiles;
}
module.exports = scanPhoto;

View file

@ -0,0 +1,62 @@
// api_v1/scanner/scanPhotoCartella.js
const scanCartella = require('./scanCartella');
const scanPhotoSingle = require('./scanPhotoSingle');
const { log } = require('./logger');
async function scanPhotoCartella(
db,
user,
cartella,
absCartella,
newFiles,
updateStatusFile,
CURRENT,
TOTAL_FILES,
start,
deleteThumbsById,
deleteFromDB,
onProgress // 🔥 AGGIUNTO
) {
log(`📁 Scan cartella: ${cartella}`);
// Recupera gli ID già presenti nel DB
const rows = await db('photos')
.where({ user, cartella })
.select('id');
const idsSet = new Set(rows.map(r => r.id));
// Scansiona i file della cartella
for await (const f of scanCartella(user, cartella, absCartella, db)) {
// Aggiorna il contatore
CURRENT.value++;
// 🔥 Aggiorna stato avanzamento (vecchio sistema)
await updateStatusFile(CURRENT, TOTAL_FILES, start);
// 🔥 Aggiorna stato avanzamento (nuovo sistema)
if (onProgress) {
await onProgress();
}
// Processa il file
const meta = await scanPhotoSingle(db, user, cartella, f, newFiles);
// Anche se identico, NON è orfano
idsSet.delete(f.id);
}
// Gestione orfani
for (const orphanId of idsSet) {
log(`🔴 Orfano: ${orphanId}`);
await deleteThumbsById(orphanId);
await deleteFromDB(orphanId, user);
}
}
module.exports = scanPhotoCartella;

View file

@ -0,0 +1,419 @@
// api_v1/scanner/scanPhotoSingle.js
const path = require('path');
const fsp = require('fs/promises');
const processFile = require('./processFile');
const {
enqueueGeocodeJob,
} = require('./geocodeQueue');
const { sha256 } = require('./utils');
const { log } = require('./logger');
const { WEB_ROOT } = require('../config');
function normCountryCode(value) {
if (!value) return undefined;
const s = String(value).trim().toUpperCase();
return s.length ? s : undefined;
}
function normalizeLocation(location) {
if (!location) return null;
const countryCode = normCountryCode(
location.country_code ||
location.countryCode ||
location.isoCountryCode
);
const countyCode =
location.county_code ||
location.countyCode ||
undefined;
return {
...location,
// Paese
country: location.country || undefined,
country_code: countryCode,
// County / contea
county: location.county || undefined,
county_code: countyCode,
// Campi già esistenti
region: location.region || undefined,
city: location.city || undefined,
postcode: location.postcode || undefined,
address: location.address || undefined,
timezone: location.timezone || undefined,
time: location.time || undefined,
};
}
// ---------------------------------------------------------
// Controlla se un file esiste fisicamente
// ---------------------------------------------------------
async function fileExists(filePath) {
try {
await fsp.access(filePath);
return true;
} catch {
return false;
}
}
// ---------------------------------------------------------
// Controlla i thumbnail fisici
//
// FOTO -> min + avg + cast
// VIDEO -> min + avg
// ---------------------------------------------------------
async function thumbnailsExist(user, cartella, f) {
const thumbBase = path.join(
WEB_ROOT,
'photos',
user,
'thumbs',
cartella,
path.dirname(f.relPath)
);
const parsed = path.parse(f.name);
const baseName = parsed.name;
const thumbMin = path.join(
thumbBase,
`${baseName}_min.jpg`
);
const thumbAvg = path.join(
thumbBase,
`${baseName}_avg.jpg`
);
const isVideo = [
'.mp4',
'.mov',
'.m4v'
].includes(f.ext.toLowerCase());
// I video NON hanno cast
if (isVideo) {
return (
await fileExists(thumbMin) &&
await fileExists(thumbAvg)
);
}
// Le foto hanno min + avg + cast
const thumbCast = path.join(
thumbBase,
`${baseName}_cast.jpg`
);
return (
await fileExists(thumbMin) &&
await fileExists(thumbAvg) &&
await fileExists(thumbCast)
);
}
async function scanPhotoSingle(
db,
user,
cartella,
f,
newFiles,
prefix = ''
) {
const fileName = f.name;
const id = f.id;
const st = f.stat;
// Recupero precedente SENZA contentHash
let prev = await db('photos')
.select(
'id',
'size_bytes',
'mtimeMs',
'fast_hash',
'path'
)
.where({ id })
.first();
// Hash veloce basato su size + mtime
const fastHash = sha256(
`${st.size}-${st.mtimeMs}`
);
// ---------------------------------------------------------
// PATH CAMBIATO = NUOVO FILE
// ---------------------------------------------------------
if (prev && prev.path !== f.path) {
log(
`${prefix} 🟢 Nuovo/Modificato (path changed): ${fileName}`
);
prev = null;
}
// ---------------------------------------------------------
// NUOVO FILE
// ---------------------------------------------------------
if (!prev) {
log(`${prefix} 🟢 Nuovo: ${fileName}`);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
// Normalizza location prima di salvarla e inviarla
meta.location = normalizeLocation(
meta.location
);
log(
`🌍 [LOCATION NEW] ${fileName} ` +
`location=${JSON.stringify(meta.location)}`
);
await db('photos').insert({
id: meta.id,
user,
cartella,
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
cast: meta.cast,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
// Qui ora il JSON può contenere:
// country, country_code, county, county_code,
// region, city, ecc.
location: meta.location
? JSON.stringify(meta.location)
: null,
fast_hash: fastHash,
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
});
await enqueueGeocodeJob({
entityType: 'photo',
entityId: meta.id,
lat: meta.gps?.lat,
lon: meta.gps?.lng,
});
newFiles.push(meta);
return meta;
}
// ---------------------------------------------------------
// IDENTICO (fastHash uguale)
//
// Se il file è identico:
// - se tutti i thumbnail esistono -> non fare niente
// - se manca anche un solo thumbnail -> riprocessa
//
// FOTO -> min + avg + cast
// VIDEO -> min + avg
// ---------------------------------------------------------
if (prev.fast_hash === fastHash) {
const thumbsOk = await thumbnailsExist(
user,
cartella,
f
);
if (thumbsOk) {
await db('photos')
.where({ id })
.update({
path: f.path,
size_bytes: st.size,
mtimeMs: st.mtimeMs,
fast_hash: fastHash,
// Anche un file identico aggiorna updated_at
updated_at: new Date().toISOString(),
});
return null;
}
// -------------------------------------------------------
// File identico ma thumbnail mancanti
// -------------------------------------------------------
log(
`${prefix} 🟡 File identico ma thumbnail mancanti: ${fileName}`
);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
// Normalizza location
meta.location = normalizeLocation(
meta.location
);
log(
`🌍 [LOCATION THUMBS REBUILD] ${fileName} ` +
`location=${JSON.stringify(meta.location)}`
);
await db('photos')
.where({ id })
.update({
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
cast: meta.cast,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
location: meta.location
? JSON.stringify(meta.location)
: null,
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
await enqueueGeocodeJob({
entityType: 'photo',
entityId: meta.id,
lat: meta.gps?.lat,
lon: meta.gps?.lng,
});
newFiles.push(meta);
return meta;
}
// ---------------------------------------------------------
// MODIFICATO (fastHash diverso)
// ---------------------------------------------------------
log(`${prefix} 🟠 Modificato: ${fileName}`);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
// Normalizza location prima di salvarla e inviarla
meta.location = normalizeLocation(
meta.location
);
log(
`🌍 [LOCATION UPDATED] ${fileName} ` +
`location=${JSON.stringify(meta.location)}`
);
await db('photos')
.where({ id })
.update({
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
// Qui ora il JSON può contenere:
// country, country_code, county, county_code,
// region, city, ecc.
location: meta.location
? JSON.stringify(meta.location)
: null,
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
await enqueueGeocodeJob({
entityType: 'photo',
entityId: meta.id,
lat: meta.gps?.lat,
lon: meta.gps?.lng,
});
newFiles.push(meta);
return meta;
}
module.exports = scanPhotoSingle;

View file

@ -0,0 +1,395 @@
// api_v1/scanner/scanPhotoSingle.js
const path = require('path');
const fsp = require('fs/promises');
const processFile = require('./processFile');
const { sha256 } = require('./utils');
const { log } = require('./logger');
const { WEB_ROOT } = require('../config');
function normCountryCode(value) {
if (!value) return undefined;
const s = String(value).trim().toUpperCase();
return s.length ? s : undefined;
}
function normalizeLocation(location) {
if (!location) return null;
const countryCode = normCountryCode(
location.country_code ||
location.countryCode ||
location.isoCountryCode
);
const countyCode =
location.county_code ||
location.countyCode ||
undefined;
return {
...location,
// Paese
country: location.country || undefined,
country_code: countryCode,
// County / contea
county: location.county || undefined,
county_code: countyCode,
// Campi già esistenti
region: location.region || undefined,
city: location.city || undefined,
postcode: location.postcode || undefined,
address: location.address || undefined,
timezone: location.timezone || undefined,
time: location.time || undefined,
};
}
// ---------------------------------------------------------
// Controlla se un file esiste fisicamente
// ---------------------------------------------------------
async function fileExists(filePath) {
try {
await fsp.access(filePath);
return true;
} catch {
return false;
}
}
// ---------------------------------------------------------
// Controlla i thumbnail fisici
//
// FOTO -> min + avg + cast
// VIDEO -> min + avg
// ---------------------------------------------------------
async function thumbnailsExist(user, cartella, f) {
const thumbBase = path.join(
WEB_ROOT,
'photos',
user,
'thumbs',
cartella,
path.dirname(f.relPath)
);
const parsed = path.parse(f.name);
const baseName = parsed.name;
const thumbMin = path.join(
thumbBase,
`${baseName}_min.jpg`
);
const thumbAvg = path.join(
thumbBase,
`${baseName}_avg.jpg`
);
const isVideo = [
'.mp4',
'.mov',
'.m4v'
].includes(f.ext.toLowerCase());
// I video NON hanno cast
if (isVideo) {
return (
await fileExists(thumbMin) &&
await fileExists(thumbAvg)
);
}
// Le foto hanno min + avg + cast
const thumbCast = path.join(
thumbBase,
`${baseName}_cast.jpg`
);
return (
await fileExists(thumbMin) &&
await fileExists(thumbAvg) &&
await fileExists(thumbCast)
);
}
async function scanPhotoSingle(
db,
user,
cartella,
f,
newFiles,
prefix = ''
) {
const fileName = f.name;
const id = f.id;
const st = f.stat;
// Recupero precedente SENZA contentHash
let prev = await db('photos')
.select(
'id',
'size_bytes',
'mtimeMs',
'fast_hash',
'path'
)
.where({ id })
.first();
// Hash veloce basato su size + mtime
const fastHash = sha256(
`${st.size}-${st.mtimeMs}`
);
// ---------------------------------------------------------
// PATH CAMBIATO = NUOVO FILE
// ---------------------------------------------------------
if (prev && prev.path !== f.path) {
log(
`${prefix} 🟢 Nuovo/Modificato (path changed): ${fileName}`
);
prev = null;
}
// ---------------------------------------------------------
// NUOVO FILE
// ---------------------------------------------------------
if (!prev) {
log(`${prefix} 🟢 Nuovo: ${fileName}`);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
// Normalizza location prima di salvarla e inviarla
meta.location = normalizeLocation(
meta.location
);
log(
`🌍 [LOCATION NEW] ${fileName} ` +
`location=${JSON.stringify(meta.location)}`
);
await db('photos').insert({
id: meta.id,
user,
cartella,
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
cast: meta.cast,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
// Qui ora il JSON può contenere:
// country, country_code, county, county_code,
// region, city, ecc.
location: meta.location
? JSON.stringify(meta.location)
: null,
fast_hash: fastHash,
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
});
newFiles.push(meta);
return meta;
}
// ---------------------------------------------------------
// IDENTICO (fastHash uguale)
//
// Se il file è identico:
// - se tutti i thumbnail esistono -> non fare niente
// - se manca anche un solo thumbnail -> riprocessa
//
// FOTO -> min + avg + cast
// VIDEO -> min + avg
// ---------------------------------------------------------
if (prev.fast_hash === fastHash) {
const thumbsOk = await thumbnailsExist(
user,
cartella,
f
);
if (thumbsOk) {
await db('photos')
.where({ id })
.update({
path: f.path,
size_bytes: st.size,
mtimeMs: st.mtimeMs,
fast_hash: fastHash,
// Anche un file identico aggiorna updated_at
updated_at: new Date().toISOString(),
});
return null;
}
// -------------------------------------------------------
// File identico ma thumbnail mancanti
// -------------------------------------------------------
log(
`${prefix} 🟡 File identico ma thumbnail mancanti: ${fileName}`
);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
// Normalizza location
meta.location = normalizeLocation(
meta.location
);
log(
`🌍 [LOCATION THUMBS REBUILD] ${fileName} ` +
`location=${JSON.stringify(meta.location)}`
);
await db('photos')
.where({ id })
.update({
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
cast: meta.cast,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
location: meta.location
? JSON.stringify(meta.location)
: null,
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
newFiles.push(meta);
return meta;
}
// ---------------------------------------------------------
// MODIFICATO (fastHash diverso)
// ---------------------------------------------------------
log(`${prefix} 🟠 Modificato: ${fileName}`);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
// Normalizza location prima di salvarla e inviarla
meta.location = normalizeLocation(
meta.location
);
log(
`🌍 [LOCATION UPDATED] ${fileName} ` +
`location=${JSON.stringify(meta.location)}`
);
await db('photos')
.where({ id })
.update({
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
// Qui ora il JSON può contenere:
// country, country_code, county, county_code,
// region, city, ecc.
location: meta.location
? JSON.stringify(meta.location)
: null,
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
newFiles.push(meta);
return meta;
}
module.exports = scanPhotoSingle;

View file

@ -0,0 +1,207 @@
// api_v1/scanner/scanPhotoSingle.js
const processFile = require('./processFile');
const { sha256 } = require('./utils');
const { log } = require('./logger');
function normCountryCode(value) {
if (!value) return undefined;
const s = String(value).trim().toUpperCase();
return s.length ? s : undefined;
}
function normalizeLocation(location) {
if (!location) return null;
const countryCode = normCountryCode(
location.country_code ||
location.countryCode ||
location.isoCountryCode
);
const countyCode =
location.county_code ||
location.countyCode ||
undefined;
return {
...location,
// Paese
country: location.country || undefined,
country_code: countryCode,
// County / contea
county: location.county || undefined,
county_code: countyCode,
// Campi già esistenti
region: location.region || undefined,
city: location.city || undefined,
postcode: location.postcode || undefined,
address: location.address || undefined,
timezone: location.timezone || undefined,
time: location.time || undefined,
};
}
async function scanPhotoSingle(db, user, cartella, f, newFiles, prefix = '') {
const fileName = f.name;
const id = f.id;
const st = f.stat;
// Recupero precedente SENZA contentHash
let prev = await db('photos')
.select('id', 'size_bytes', 'mtimeMs', 'fast_hash', 'path')
.where({ id })
.first();
// Hash veloce basato su size + mtime
const fastHash = sha256(`${st.size}-${st.mtimeMs}`);
// ---------------------------------------------------------
// PATH CAMBIATO = NUOVO FILE
// ---------------------------------------------------------
if (prev && prev.path !== f.path) {
log(`${prefix} 🟢 Nuovo/Modificato (path changed): ${fileName}`);
prev = null;
}
// ---------------------------------------------------------
// NUOVO FILE
// ---------------------------------------------------------
if (!prev) {
log(`${prefix} 🟢 Nuovo: ${fileName}`);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
// Normalizza location prima di salvarla e inviarla
meta.location = normalizeLocation(meta.location);
log(
`🌍 [LOCATION NEW] ${fileName} ` +
`location=${JSON.stringify(meta.location)}`
);
await db('photos').insert({
id: meta.id,
user,
cartella,
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
cast: meta.cast,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
// Qui ora il JSON può contenere:
// country, country_code, county, county_code, region, city, ecc.
location: meta.location ? JSON.stringify(meta.location) : null,
fast_hash: fastHash,
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
});
newFiles.push(meta);
return meta;
}
// ---------------------------------------------------------
// IDENTICO (fastHash uguale)
// ---------------------------------------------------------
if (prev.fast_hash === fastHash) {
await db('photos')
.where({ id })
.update({
path: f.path,
size_bytes: st.size,
mtimeMs: st.mtimeMs,
fast_hash: fastHash,
// Anche un file identico aggiorna updated_at
updated_at: new Date().toISOString(),
});
return null;
}
// ---------------------------------------------------------
// MODIFICATO (fastHash diverso)
// ---------------------------------------------------------
log(`${prefix} 🟠 Modificato: ${fileName}`);
const meta = await processFile(
user,
cartella,
f.relPath,
f.absPath,
f.ext,
st
);
meta.id = id;
meta.path = f.path;
// Normalizza location prima di salvarla e inviarla
meta.location = normalizeLocation(meta.location);
log(
`🌍 [LOCATION UPDATED] ${fileName} ` +
`location=${JSON.stringify(meta.location)}`
);
await db('photos')
.where({ id })
.update({
name: meta.name,
path: meta.path,
thub1: meta.thub1,
thub2: meta.thub2,
mime_type: meta.mime_type,
width: meta.width,
height: meta.height,
rotation: meta.rotation,
size_bytes: meta.size_bytes,
mtimeMs: meta.mtimeMs,
duration_ms: meta.duration_ms,
taken_at: meta.taken_at,
data: meta.data,
lat: meta.gps?.lat ?? null,
lon: meta.gps?.lng ?? null,
alt: meta.gps?.alt ?? null,
// Qui ora il JSON può contenere:
// country, country_code, county, county_code, region, city, ecc.
location: meta.location ? JSON.stringify(meta.location) : null,
fast_hash: fastHash,
updated_at: new Date().toISOString(),
});
newFiles.push(meta);
return meta;
}
module.exports = scanPhotoSingle;

View file

@ -0,0 +1,161 @@
// api_v1/scanner/scanPhotosUser.js
const path = require('path');
const fsp = require('fs/promises');
const scanPhoto = require('./scanPhoto');
const { log } = require('./logger');
const { WEB_ROOT, SUPPORTED_EXTS } = require('../config');
const writeJsonAtomic = require('./writeJsonAtomic');
// ---------------------------------------------------------
// ETA CALCULATOR
// ---------------------------------------------------------
function computeETA(startTime, current, total) {
if (current === 0) return 'calcolo...';
const elapsed = (Date.now() - startTime) / 1000;
const rate = current / elapsed;
const remaining = (total - current) / rate;
const m = Math.floor(remaining / 60);
const s = Math.floor(remaining % 60);
return `${m}m ${s}s`;
}
// ---------------------------------------------------------
// PRIMA PASSATA: conta TUTTI i file reali ricorsivamente
// ---------------------------------------------------------
async function countFilesUser(rootDir) {
let count = 0;
async function walk(dir) {
let entries = [];
try {
entries = await fsp.readdir(dir, { withFileTypes: true });
} catch {
return;
}
for (const e of entries) {
const abs = path.join(dir, e.name);
if (e.isDirectory()) {
await walk(abs);
} else {
const ext = path.extname(e.name).toLowerCase();
if (SUPPORTED_EXTS.has(ext)) {
count++;
}
}
}
}
await walk(rootDir);
return count;
}
// ---------------------------------------------------------
// SECONDA PASSATA: scansiona SOLO le cartelle vere
// ---------------------------------------------------------
async function scanPhotosUser(userName, db) {
log(`🔵 Inizio scan TUTTE le cartelle per user=${userName}`);
const photosRoot = path.resolve(__dirname, '..', '..', WEB_ROOT, 'photos');
const userDir = path.join(photosRoot, userName, 'original');
const statusPath = path.join(photosRoot, 'scan_status.json');
let entries = [];
try {
entries = await fsp.readdir(userDir, { withFileTypes: true });
} catch {
log(`❌ Nessuna directory per utente ${userName}`);
return [];
}
// Filtra SOLO cartelle vere dentro "original"
const folders = entries
.filter(e => e.isDirectory())
.map(e => e.name);
// ---------------------------------------------------------
// RIMOZIONE CARTELLE CANCELLATE DAL FILESYSTEM
// ---------------------------------------------------------
const createDeleteFolderFunctions = require('./deleteFolder');
const { deleteFolderForUser } = createDeleteFolderFunctions(db);
const dbFolders = await db('photos')
.where({ user: userName })
.distinct('cartella')
.pluck('cartella');
for (const dbFolder of dbFolders) {
if (!folders.includes(dbFolder)) {
log(`🗑️ Cartella rimossa dal filesystem: ${dbFolder}`);
await deleteFolderForUser(userName, dbFolder);
}
}
// PRIMA PASSATA: conta i file reali
const TOTAL_FILES = await countFilesUser(userDir);
const CURRENT = { value: 0 };
const start = Date.now();
log(`📊 File totali da scansionare: ${TOTAL_FILES}`);
// Stato iniziale
await writeJsonAtomic(statusPath, {
current: 0,
total: TOTAL_FILES,
percent: 0,
eta: 'calcolo...'
});
const allNewFiles = [];
// SECONDA PASSATA: UNA SOLA SCANSIONE PER CARTELLA
for (const cartella of folders) {
log(`📁 Scan cartella utente: ${cartella}`);
const newFiles = await scanPhoto(
cartella,
userName,
db,
CURRENT,
TOTAL_FILES,
start,
async () => {
// Callback di aggiornamento progresso
await writeJsonAtomic(statusPath, {
current: CURRENT.value,
total: TOTAL_FILES,
percent: TOTAL_FILES > 0
? Math.round((CURRENT.value / TOTAL_FILES) * 100)
: 100,
eta: computeETA(start, CURRENT.value, TOTAL_FILES)
});
}
);
if (newFiles?.length) {
allNewFiles.push(...newFiles);
}
}
// Stato finale
await writeJsonAtomic(statusPath, {
current: TOTAL_FILES,
total: TOTAL_FILES,
percent: 100,
eta: '0m 0s'
});
log(`🟣 Scan COMPLETATO per user=${userName}. Nuovi file totali: ${allNewFiles.length}`);
return allNewFiles;
}
module.exports = scanPhotosUser;

View file

@ -0,0 +1,25 @@
// api_v1/scanner/scanUser.js
const scanPhotosUser = require('./scanPhotosUser');
const scanActivitiesUser = require('./scanActivitiesUser');
const { log } = require('./logger');
async function scanUser(userName, db) {
log(`🔵 Inizio scan COMPLETO per user=${userName}`);
// 🔥 SCAN FOTO
const newPhotos = await scanPhotosUser(userName, db);
// 🔥 SCAN ATTIVITÀ
const newActivities = await scanActivitiesUser(userName, db);
log(`🟣 Scan COMPLETATO per user=${userName}`);
log(` 📸 Nuove foto: ${newPhotos.length}`);
log(` 🏃‍♂️ Nuove attività: ${newActivities.length}`);
return {
photos: newPhotos,
activities: newActivities
};
}
module.exports = scanUser;

335
api_v1/scanner/thumbs.js Normal file
View file

@ -0,0 +1,335 @@
// api_v1/scanner/thumbs.js
const fs = require('fs');
const sharp = require('sharp');
const { exec } = require('child_process');
const path = require('path');
/*
* ============================================================
* CONFIGURAZIONE IMMAGINE CAST
* ============================================================
*/
/*
* Se il file originale è JPEG e pesa meno di questa soglia,
* il file _cast.jpg sarà una copia identica dell'originale.
*
* 500 * 1024 = 512000 byte.
*/
const CAST_JPEG_COPY_LIMIT_BYTES = 500 * 1024;
/*
* Dimensioni massime dell'immagine Cast.
*
* L'immagine viene sempre contenuta dentro questo rettangolo,
* senza crop e senza ingrandimento.
*/
const CAST_MAX_WIDTH = 1920;
const CAST_MAX_HEIGHT = 1080;
/*
* Qualità JPEG usata quando è necessaria la conversione.
*/
const CAST_JPEG_QUALITY = 85;
function deriveCastPathFromAvg(thumbAvgPath) {
if (!thumbAvgPath) {
return null;
}
/*
* Caso standard:
*
* IMG_0241_avg.jpg
* diventa:
* IMG_0241_cast.jpg
*/
if (thumbAvgPath.toLowerCase().endsWith('_avg.jpg')) {
return thumbAvgPath.replace(
/_avg\.jpg$/i,
'_cast.jpg'
);
}
/*
* Fallback generico.
*/
const ext = path.extname(thumbAvgPath);
const base = thumbAvgPath.slice(
0,
-ext.length
);
return `${base}_cast.jpg`;
}
/*
* Video: crea soltanto AVG e MIN.
*/
function createVideoThumbnail(
videoPath,
thumbMinPath,
thumbAvgPath
) {
return new Promise(resolve => {
const cmd = `
ffmpeg -y -i "${videoPath}" -ss 00:00:01.000 -vframes 1 "${thumbAvgPath}" &&
ffmpeg -y -i "${thumbAvgPath}" -vf "scale=100:-1" "${thumbMinPath}"
`;
exec(cmd, error => {
if (error) {
console.error(
'[THUMB VIDEO] errore creazione thumbnail',
{
videoPath,
thumbMinPath,
thumbAvgPath,
error: error.message,
}
);
} else {
console.log(
'[THUMB VIDEO] thumbnails create',
{
videoPath,
thumbMinPath,
thumbAvgPath,
}
);
}
/*
* Mantiene il comportamento originale:
* la Promise viene risolta anche in caso di errore.
*/
resolve();
});
});
}
/*
* Crea la versione CAST dell'immagine.
*
* Regole:
*
* 1. Se l'originale è JPEG e pesa meno di 500 KB:
* copia byte per byte l'originale nel file _cast.jpg.
*
* 2. In tutti gli altri casi:
* converte l'immagine in JPEG qualità 85.
*
* 3. Se l'immagine supera 1920x1080:
* viene ridimensionata per stare dentro 1920x1080.
*
* 4. Se l'immagine è già più piccola:
* mantiene le sue dimensioni, senza ingrandirla.
*
* 5. Non viene mai effettuato crop.
*/
async function createCastImage(
filePath,
finalCastPath
) {
const sourceStat =
await fs.promises.stat(filePath);
const metadata =
await sharp(filePath).metadata();
/*
* metadata.format === "jpeg" riconosce il contenuto reale
* dell'immagine, indipendentemente dall'estensione .jpg o .jpeg.
*/
const sourceIsJpeg =
metadata.format === 'jpeg';
const sourceIsUnderCopyLimit =
sourceStat.size <
CAST_JPEG_COPY_LIMIT_BYTES;
/*
* JPEG sotto 500 KB:
* copia identica dell'originale.
*
* Non viene effettuata:
*
* - ricompressione;
* - rotazione;
* - modifica dei metadati;
* - modifica delle dimensioni.
*/
if (
sourceIsJpeg &&
sourceIsUnderCopyLimit
) {
await fs.promises.copyFile(
filePath,
finalCastPath
);
const castStat =
await fs.promises.stat(finalCastPath);
console.log(
'[THUMB CAST] originale JPEG copiato',
{
source: filePath,
destination: finalCastPath,
sourceBytes: sourceStat.size,
castBytes: castStat.size,
copyLimitBytes:
CAST_JPEG_COPY_LIMIT_BYTES,
sourceFormat: metadata.format,
sourceWidth: metadata.width || null,
sourceHeight: metadata.height || null,
}
);
return;
}
/*
* Tutti gli altri casi:
*
* - JPEG da almeno 500 KB;
* - PNG;
* - WebP;
* - HEIC/HEIF, se supportato da Sharp;
* - TIFF;
* - altri formati decodificabili da Sharp.
*
* L'immagine viene convertita in JPEG qualità 85.
*
* withoutEnlargement: true garantisce che un'immagine più
* piccola di 1920x1080 mantenga le proprie dimensioni.
*/
await sharp(filePath)
.rotate()
.resize({
width: CAST_MAX_WIDTH,
height: CAST_MAX_HEIGHT,
fit: 'inside',
withoutEnlargement: true,
fastShrinkOnLoad: true,
})
.jpeg({
quality: CAST_JPEG_QUALITY,
progressive: false,
optimizeCoding: true,
})
.toFile(finalCastPath);
const castStat =
await fs.promises.stat(finalCastPath);
const castMetadata =
await sharp(finalCastPath).metadata();
console.log(
'[THUMB CAST] immagine convertita',
{
source: filePath,
destination: finalCastPath,
sourceBytes: sourceStat.size,
castBytes: castStat.size,
sourceFormat:
metadata.format || null,
sourceWidth:
metadata.width || null,
sourceHeight:
metadata.height || null,
castFormat:
castMetadata.format || null,
castWidth:
castMetadata.width || null,
castHeight:
castMetadata.height || null,
quality: CAST_JPEG_QUALITY,
maxWidth: CAST_MAX_WIDTH,
maxHeight: CAST_MAX_HEIGHT,
reason: sourceIsJpeg
? 'JPEG sopra o uguale alla soglia'
: 'formato originale non JPEG',
}
);
}
/*
* Foto: crea MIN, AVG e CAST.
*/
async function createThumbnails(
filePath,
thumbMinPath,
thumbAvgPath,
thumbCastPath
) {
try {
const finalCastPath =
thumbCastPath ||
deriveCastPathFromAvg(thumbAvgPath);
/*
* 1. MIN: massimo 100x100.
*/
await sharp(filePath)
.rotate()
.resize({
width: 100,
height: 100,
fit: 'inside',
withoutEnlargement: true,
})
.jpeg({
quality: 75,
progressive: false,
optimizeCoding: true,
})
.toFile(thumbMinPath);
/*
* 2. AVG: larghezza massima 400px.
*/
await sharp(filePath)
.rotate()
.resize({
width: 400,
fit: 'inside',
withoutEnlargement: true,
})
.jpeg({
quality: 78,
progressive: false,
optimizeCoding: true,
})
.toFile(thumbAvgPath);
/*
* 3. CAST.
*/
if (finalCastPath) {
await createCastImage(
filePath,
finalCastPath
);
}
} catch (err) {
console.error(
'[THUMB] errore creazione thumbnails',
{
filePath,
thumbMinPath,
thumbAvgPath,
thumbCastPath,
error: err.message,
stack: err.stack,
}
);
}
}
module.exports = {
createVideoThumbnail,
createThumbnails,
};

View file

@ -0,0 +1,96 @@
// api_v1/scanner/thumbs.js
const sharp = require('sharp');
const { exec } = require('child_process');
const path = require('path');
function deriveCastPathFromAvg(thumbAvgPath) {
if (!thumbAvgPath) return null;
// Caso standard: IMG_0241_avg.jpg -> IMG_0241_cast.jpg
if (thumbAvgPath.toLowerCase().endsWith('_avg.jpg')) {
return thumbAvgPath.replace(/_avg\.jpg$/i, '_cast.jpg');
}
// Fallback generico
const ext = path.extname(thumbAvgPath);
const base = thumbAvgPath.slice(0, -ext.length);
return `${base}_cast.jpg`;
}
// Video: lascia solo avg + min
function createVideoThumbnail(videoPath, thumbMinPath, thumbAvgPath) {
return new Promise((resolve) => {
const cmd = `
ffmpeg -y -i "${videoPath}" -ss 00:00:01.000 -vframes 1 "${thumbAvgPath}" &&
ffmpeg -y -i "${thumbAvgPath}" -vf "scale=100:-1" "${thumbMinPath}"
`;
exec(cmd, () => resolve());
});
}
// Foto: min + avg + cast
async function createThumbnails(filePath, thumbMinPath, thumbAvgPath, thumbCastPath) {
try {
const finalCastPath = thumbCastPath || deriveCastPathFromAvg(thumbAvgPath);
// 1) MIN: 100px
await sharp(filePath)
.rotate()
.resize({
width: 100,
height: 100,
fit: 'inside',
withoutEnlargement: true,
})
.jpeg({
quality: 75,
progressive: false,
optimizeCoding: true,
})
.toFile(thumbMinPath);
// 2) AVG: 400px
await sharp(filePath)
.rotate()
.resize({
width: 400,
fit: 'inside',
withoutEnlargement: true,
})
.jpeg({
quality: 78,
progressive: false,
optimizeCoding: true,
})
.toFile(thumbAvgPath);
// 3) CAST: max 4K TV, dentro 3840x2160, senza crop
// 3) CAST: metà del 4K → 1920x1080, senza crop
if (finalCastPath) {
await sharp(filePath)
.rotate()
.resize({
width: 1920,
height: 1080,
fit: 'inside',
withoutEnlargement: true,
fastShrinkOnLoad: true,
})
.jpeg({
quality: 85,
progressive: false,
optimizeCoding: true,
})
.toFile(finalCastPath);
}
} catch (err) {
console.error('Errore creazione thumbnails:', err.message, filePath);
}
}
module.exports = {
createVideoThumbnail,
createThumbnails,
};

41
api_v1/scanner/utils.js Normal file
View file

@ -0,0 +1,41 @@
const path = require('path');
const crypto = require('crypto');
function toPosix(p) {
return p.split(path.sep).join('/');
}
function sha256(s) {
return crypto.createHash('sha256').update(s).digest('hex');
}
function inferMimeFromExt(ext) {
switch (ext.toLowerCase()) {
case '.jpg':
case '.jpeg': return 'image/jpeg';
case '.png': return 'image/png';
case '.webp': return 'image/webp';
case '.heic':
case '.heif': return 'image/heic';
case '.mp4': return 'video/mp4';
case '.mov': return 'video/quicktime';
case '.m4v': return 'video/x-m4v';
default: return 'application/octet-stream';
}
}
function parseExifDateUtc(s) {
if (!s) return null;
const re = /^(\d{4}):(\d{2}):(\d{2}) (\d{2}):(\d{2}):(\d{2})$/;
const m = re.exec(s);
if (!m) return null;
const dt = new Date(Date.UTC(+m[1], +m[2]-1, +m[3], +m[4], +m[5], +m[6]));
return dt.toISOString();
}
module.exports = {
toPosix,
sha256,
inferMimeFromExt,
parseExifDateUtc
};

17
api_v1/scanner/video.js Normal file
View file

@ -0,0 +1,17 @@
const { exec } = require('child_process');
function probeVideo(videoPath) {
return new Promise((resolve) => {
const cmd = `ffprobe -v quiet -print_format json -show_format -show_streams "${videoPath}"`;
exec(cmd, (err, stdout) => {
if (err) return resolve({});
try {
resolve(JSON.parse(stdout));
} catch {
resolve({});
}
});
});
}
module.exports = { probeVideo };

View file

@ -0,0 +1,17 @@
// api_v1/scanner/writeJsonAtomic.js
const path = require('path');
const fsp = require('fs/promises');
async function writeJsonAtomic(filePath, data) {
const tmpPath = `${filePath}.${process.pid}.${Date.now()}.${Math.random()
.toString(16)
.slice(2)}.tmp`;
const json = JSON.stringify(data, null, 2);
await fsp.mkdir(path.dirname(filePath), { recursive: true });
await fsp.writeFile(tmpPath, json, 'utf8');
await fsp.rename(tmpPath, filePath);
}
module.exports = writeJsonAtomic;

51
api_v1/tools.js Normal file
View file

@ -0,0 +1,51 @@
/**
* Required libraries
*/
const bcrypt = require('bcrypt')
const readLine = require('readline')
const async = require('async')
// Password hash method
const hashPassword = plain => bcrypt.hashSync(plain, 8)
// Ask user password method
function askPassword(question) {
return new Promise((resolve, reject) => {
const rl = readLine.createInterface({
input: process.stdin,
output: process.stdout
})
rl.question(question, answer => {
rl.close()
resolve(answer)
})
})
}
// Generate hash password method
async function generateHash() {
try {
console.log('**********************************')
console.log('** Password hash script **')
console.log('**********************************')
const passwordAnswer = await askPassword(
'Please give me a password to hash: '
)
if (passwordAnswer != '') {
const hashedPassword = hashPassword(passwordAnswer)
const compare = bcrypt.compareSync(passwordAnswer, hashedPassword)
await console.log('Hashed password:', hashedPassword)
await console.log('Valdiation:', compare)
} else {
console.log('You need write something. Script aborted!')
}
} catch (err) {
console.log(err)
return process.exit(1)
}
}
generateHash()

159
api_v1/userAccess.js Normal file
View file

@ -0,0 +1,159 @@
const fs = require('fs');
const path = require('path');
const usersPath = path.resolve(__dirname, 'users.json');
function normalize(value) {
return String(value ?? '').trim();
}
function unique(values) {
return [
...new Set(
values
.map(normalize)
.filter(Boolean)
)
];
}
function loadUsers() {
const raw = fs.readFileSync(usersPath, 'utf8');
const data = JSON.parse(raw);
if (!data || !Array.isArray(data.users)) {
throw new Error('Formato users.json non valido');
}
return data.users;
}
function findUserByName(userName) {
const requestedName = normalize(userName).toLowerCase();
if (!requestedName) {
return null;
}
return (
loadUsers().find((user) => {
return normalize(user.name).toLowerCase() === requestedName;
}) || null
);
}
function getUserAccess(userName) {
const user = findUserByName(userName);
if (!user) {
return null;
}
const devices = Array.isArray(user.devices)
? unique(user.devices)
: [];
const usersR = Array.isArray(user.usersR)
? unique(user.usersR)
: [];
const usersW = Array.isArray(user.usersW)
? unique(user.usersW)
: [];
return {
id: user.id,
name: normalize(user.name),
devices,
usersR,
usersW,
// Contenuti leggibili nella galleria
readableSources: unique([
...usersR,
...devices
]),
// Contenuti modificabili
writableSources: unique([
...usersW,
...devices
])
};
}
function getReadableSources(userName) {
const access = getUserAccess(userName);
return access
? access.readableSources
: [];
}
function getWritableSources(userName) {
const access = getUserAccess(userName);
return access
? access.writableSources
: [];
}
function filterReadableSources(userName, requestedSources) {
const readableSources = getReadableSources(userName);
if (!requestedSources) {
return readableSources;
}
const requested = Array.isArray(requestedSources)
? requestedSources
: [requestedSources];
const readableMap = new Map(
readableSources.map((source) => [
source.toLowerCase(),
source
])
);
return unique(requested)
.map((source) => {
return readableMap.get(source.toLowerCase());
})
.filter(Boolean);
}
function canRead(userName, sourceName) {
const source = normalize(sourceName).toLowerCase();
if (!source) {
return false;
}
return getReadableSources(userName).some((allowedSource) => {
return allowedSource.toLowerCase() === source;
});
}
function canWrite(userName, sourceName) {
const source = normalize(sourceName).toLowerCase();
if (!source) {
return false;
}
return getWritableSources(userName).some((allowedSource) => {
return allowedSource.toLowerCase() === source;
});
}
module.exports = {
loadUsers,
findUserByName,
getUserAccess,
getReadableSources,
getWritableSources,
filterReadableSources,
canRead,
canWrite
};

31
api_v1/users.json Normal file
View file

@ -0,0 +1,31 @@
{
"users": [
{
"id": 1,
"name": "Admin",
"email": "admin@gmail.com",
"password": "$2b$08$g0UWN6RnN7e.8rX3fuXSSOSJDTvucu./0FAU.yXp0wx4SJXyeaU3.",
"devices": [],
"usersR": ["Common", "Fabio", "Jessica", "Fabio13TPro", "Fabio11TPro"],
"usersW": ["Common", "Fabio", "Jessica", "Fabio13TPro", "Fabio11TPro"]
},
{
"id": 2,
"name": "Fabio",
"email": "fabio@gmail.com",
"password": "$2b$08$g0UWN6RnN7e.8rX3fuXSSOSJDTvucu./0FAU.yXp0wx4SJXyeaU3.",
"devices": ["Fabio13TPro", "Fabio11TPro"],
"usersR": ["Common", "Fabio"],
"usersW": ["Fabio"]
},
{
"id": 3,
"name": "Jessica",
"email": "jessie@libero.it",
"password": "$2b$08$g0UWN6RnN7e.8rX3fuXSSOSJDTvucu./0FAU.yXp0wx4SJXyeaU3.",
"devices": [],
"usersR": ["Common", "Jessica"],
"usersW": ["Jessica"]
}
]
}

0
aves.db Normal file
View file

1
b/a.jpg Normal file
View file

@ -0,0 +1 @@
prova

BIN
bin/face_scan Executable file

Binary file not shown.

25
callback_server.py Normal file
View file

@ -0,0 +1,25 @@
# callback_server.py
from http.server import HTTPServer, BaseHTTPRequestHandler
import json
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
length = int(self.headers.get('Content-Length', 0))
body = self.rfile.read(length) if length else b''
try:
data = json.loads(body.decode('utf-8')) if body else None
except Exception:
data = body.decode('utf-8', errors='replace')
print('POST', self.path)
print('Headers:', dict(self.headers))
print('Body:', data)
# rispondi 200 OK
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps({'ok': True}).encode('utf-8'))
if __name__ == '__main__':
server = HTTPServer(('192.168.1.4', 8765), Handler)
print('Listening on http://192.168.1.4:8765')
server.serve_forever()

15
cast.md Normal file
View file

@ -0,0 +1,15 @@
./transcode_rk_vod.sh ../IMG_1425.MOV ../fabiocast
python3 hls_cors_server.py ../fabiocast -p 9999
catt -d 192.168.1.15 cast http://192.168.1.4:9999/master.m3u8
./transcode_rk_vod.sh \
../IMG_1425.MOV \
../fabiocast
python3 hls_cors_server.py \
../fabiocast \
-p 9999
catt -d 192.168.1.15 cast \
http://192.168.1.4:9999/master.m3u8

19
cast.sh Normal file
View file

@ -0,0 +1,19 @@
#!/usr/bin/env bash
SERVER="https://prova.patachina.it"
EMAIL="fabio@gmail.com"
PASS="master66"
ENTRY_ID="751d8a15766bf2686bc87e60b33b4a8c38befe454428e6a5a82b00f26026c0e0"
TOKEN=$(curl -s -X POST "$SERVER/auth/login" -H "Content-Type: application/json" -d "{\"email\":\"$EMAIL\",\"password\":\"$PASS\"}" | jq -r .token)
echo "JWT: $TOKEN"
RESP=$(curl -s -X POST "$SERVER/share/create" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "{\"entryId\":\"$ENTRY_ID\",\"ttlSeconds\":600,\"oneTime\":false}")
echo "create response: $RESP"
URL=$(echo "$RESP" | jq -r .url)
echo "share url: $URL"
echo "download full"
curl -v -L "$URL" -o /tmp/test.jpg
echo "test range"
curl -v -L -H "Range: bytes=0-1023" "$URL" -o /tmp/part.bin

58
cast1.sh Normal file
View file

@ -0,0 +1,58 @@
#!/usr/bin/env bash
# CONFIGURA QUI
SERVER="https://prova.patachina.it"
EMAIL="fabio@gmail.com"
PASS="master66"
ENTRY_ID="751d8a15766bf2686bc87e60b33b4a8c38befe454428e6a5a82b00f26026c0e0"
TTL=600
# FILE temporanei
OUT_FULL="/tmp/share_test_full.jpg"
OUT_PART="/tmp/share_test_part.bin"
RESP_JSON="/tmp/share_create_resp.json"
echo "1) Login..."
TOKEN=$(curl -s -X POST "$SERVER/auth/login" \
-H "Content-Type: application/json" \
-d "{\"email\":\"$EMAIL\",\"password\":\"$PASS\"}" | jq -r .token)
if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
echo "Login fallito. Controlla credenziali."
exit 1
fi
echo "JWT ottenuto."
echo "2) Creo share per entryId $ENTRY_ID (ttl=${TTL}s)..."
curl -s -X POST "$SERVER/share/create" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "{\"entryId\":\"$ENTRY_ID\",\"ttlSeconds\":$TTL,\"oneTime\":false}" \
-o "$RESP_JSON"
cat "$RESP_JSON" | jq .
URL=$(jq -r .url < "$RESP_JSON")
EXPIRES_AT=$(jq -r .expiresAt < "$RESP_JSON")
if [ -z "$URL" ] || [ "$URL" = "null" ]; then
echo "Creazione share fallita. Output:"
cat "$RESP_JSON"
exit 1
fi
echo "Share creato: $URL"
echo "Scadenza: $EXPIRES_AT"
echo "3) Scarico file completo (seguo redirect e TLS)..."
curl -v -L "$URL" -o "$OUT_FULL"
echo "Salvato in $OUT_FULL (size: $(stat -c%s "$OUT_FULL") bytes)"
echo "4) Test Range bytes=0-1023..."
curl -v -L -H "Range: bytes=0-1023" "$URL" -o "$OUT_PART"
echo "Salvato in $OUT_PART (size: $(stat -c%s "$OUT_PART") bytes)"
echo "5) Intestazioni HEAD (senza body)"
curl -I -L "$URL" || true
echo "Fatto. Controlla i file in /tmp e i log della tua app se necessario."

202
come funziona.md Normal file
View file

@ -0,0 +1,202 @@
Perfetto Fabio — ti preparo **la guida completa, chiara e pratica** per integrare il tuo **viewer Flutter Android** con il **WebSocket del tuo server Node.js**, così come funziona oggi nel tuo backend.
Ti spiego:
1. **Come si autentica il client Flutter**
2. **Come si connette al WebSocket**
3. **Quali messaggi riceve** (formato esatto)
4. **Come deve rispondere**
5. **Come integrare tutto nel viewer**
Tutto basato sul tuo backend reale.
---
# ⭐ 1. COME SI AUTENTICA IL CLIENT FLUTTER
Il tuo server usa JWT.
### Passi:
1. Flutter fa login via HTTP:
```
POST /auth/login
{
"email": "...",
"password": "..."
}
```
2. Il server risponde:
```json
{
"token": "<JWT>",
"name": "Fabio"
}
```
3. Flutter salva il token.
4. **Ogni chiamata HTTP successiva** deve avere:
```
Authorization: Bearer <JWT>
```
---
# ⭐ 2. COME SI CONNETTE AL WEBSOCKET
Il tuo server WebSocket accetta connessioni così:
```
ws://<server>:<port>/?token=<JWT>
```
Quindi in Flutter:
```dart
import 'package:web_socket_channel/web_socket_channel.dart';
final channel = WebSocketChannel.connect(
Uri.parse("ws://192.168.1.10:4000/?token=$jwt"),
);
```
### Importante:
- Devi passare **lo stesso JWT** usato per le API HTTP.
- Il server autentica il token e associa la connessione all’utente.
---
# ⭐ 3. QUALI MESSAGGI RICEVE FLUTTER (FORMATO ESATTO)
Il tuo backend invia questi eventi:
### 📌 Quando una foto viene aggiunta
```json
{
"type": "added",
"id": "12345"
}
```
### 📌 Quando una foto viene rimossa (DEL)
```json
{
"type": "removed",
"id": "12345"
}
```
### 📌 Quando una cartella viene aggiunta (ADD_DIR)
```json
{
"type": "add_dir",
"folder": "Vacanze/2024"
}
```
### 📌 Quando una cartella viene rimossa (DEL_DIR)
```json
{
"type": "del_dir",
"folder": "Vacanze/2024"
}
```
### 📌 Quando una foto viene soft-deleted / restored
```json
{
"type": "updated",
"id": "12345",
"deleted_at": "2026-03-31T10:00:00Z" // oppure null
}
```
---
# ⭐ 4. COME FLUTTER DEVE GESTIRE I MESSAGGI
Esempio completo:
```dart
channel.stream.listen((message) {
final data = jsonDecode(message);
switch (data["type"]) {
case "added":
// Scarica la foto via /photos/:id
fetchPhoto(data["id"]);
break;
case "removed":
// Rimuovi la foto dalla UI
removePhotoFromUI(data["id"]);
break;
case "updated":
// Aggiorna soft delete
updatePhotoState(data["id"], data["deleted_at"]);
break;
case "add_dir":
// Aggiorna lista cartelle
addFolderToUI(data["folder"]);
break;
case "del_dir":
// Rimuovi cartella dalla UI
removeFolderFromUI(data["folder"]);
break;
}
});
```
---
# ⭐ 5. COME DEVE RISPONDERE FLUTTER AL SERVER
Il tuo server **non richiede risposte** dal client WebSocket.
È **one-way**:
- Il server invia eventi
- Il client li riceve
- Il client aggiorna la UI
Flutter **non deve inviare nulla** al WebSocket.
---
# ⭐ 6. COME INTEGRARE TUTTO NEL VIEWER
### Al login:
- fai login HTTP
- salva il JWT
- apri il WebSocket
### All’avvio del viewer:
1. Carica tutte le foto via:
```
GET /photos
```
2. Avvia il WebSocket
3. Applica in tempo reale:
- added → aggiungi foto
- removed → rimuovi foto
- updated → aggiorna stato
- add_dir → aggiungi cartella
- del_dir → rimuovi cartella
### Dopo 2 minuti (quando WS non è più affidabile):
- usa progressive sync:
```
GET /photos/changes?since=<timestamp>
GET /photos/deleted_hard?since=<timestamp>
```
---

209
db/activities.js Normal file
View file

@ -0,0 +1,209 @@
// db/activities.js
const knex = require('./knex');
/* ============================================================
INSERT OR UPDATE ACTIVITY
============================================================ */
async function insertOrUpdateActivity(row) {
const exists = await knex('activities').where({ id: row.id }).first();
if (!exists) {
await knex('activities').insert({
id: row.id,
user: row.user,
cartella: row.cartella,
file_name: row.file_name,
file_extension: row.file_extension,
format: row.format,
display_name: row.display_name,
route_name: row.route_name,
title: row.title,
description: row.description,
notes: row.notes,
sport: row.sport,
activity_type: row.activity_type,
device_manufacturer: row.device_manufacturer,
device_model: row.device_model,
source_app: row.source_app,
start_time_millis: row.start_time_millis,
end_time_millis: row.end_time_millis,
duration_millis: row.duration_millis,
distance_meters: row.distance_meters,
elevation_gain_meters: row.elevation_gain_meters,
elevation_loss_meters: row.elevation_loss_meters,
min_elevation_meters: row.min_elevation_meters,
max_elevation_meters: row.max_elevation_meters,
file_distance_meters: row.file_distance_meters,
file_elevation_gain_meters: row.file_elevation_gain_meters,
computed_distance_meters: row.computed_distance_meters,
computed_elevation_gain_meters: row.computed_elevation_gain_meters,
average_speed_mps: row.average_speed_mps,
max_speed_mps: row.max_speed_mps,
average_pace_sec_per_km: row.average_pace_sec_per_km,
average_heart_rate: row.average_heart_rate,
max_heart_rate: row.max_heart_rate,
average_cadence: row.average_cadence,
max_cadence: row.max_cadence,
calories: row.calories,
power_avg: row.power_avg,
power_max: row.power_max,
point_count: row.point_count,
segment_count: row.segment_count,
lap_count: row.lap_count,
min_latitude: row.min_latitude,
max_latitude: row.max_latitude,
min_longitude: row.min_longitude,
max_longitude: row.max_longitude,
center_latitude: row.center_latitude,
center_longitude: row.center_longitude,
size_bytes: row.size_bytes,
last_modified_millis: row.last_modified_millis,
date_added_millis: row.date_added_millis,
date_modified_millis: row.date_modified_millis,
last_scan_millis: row.last_scan_millis,
is_favorite: row.is_favorite,
is_hidden: row.is_hidden,
rating: row.rating,
tags: row.tags ? JSON.stringify(row.tags) : null,
scan_status: row.scan_status,
scan_error: row.scan_error,
diagnostics: row.diagnostics ? JSON.stringify(row.diagnostics) : null,
location: row.location ? JSON.stringify(row.location) : null,
created_at: knex.fn.now(),
updated_at: knex.fn.now()
});
} else {
await knex('activities').where({ id: row.id }).update({
// aggiorna solo i campi che possono cambiare
file_name: row.file_name || exists.file_name,
file_extension: row.file_extension || exists.file_extension,
format: row.format || exists.format,
display_name: row.display_name || exists.display_name,
route_name: row.route_name || exists.route_name,
title: row.title || exists.title,
description: row.description || exists.description,
notes: row.notes || exists.notes,
sport: row.sport || exists.sport,
activity_type: row.activity_type || exists.activity_type,
device_manufacturer: row.device_manufacturer || exists.device_manufacturer,
device_model: row.device_model || exists.device_model,
source_app: row.source_app || exists.source_app,
start_time_millis: row.start_time_millis || exists.start_time_millis,
end_time_millis: row.end_time_millis || exists.end_time_millis,
duration_millis: row.duration_millis || exists.duration_millis,
distance_meters: row.distance_meters || exists.distance_meters,
elevation_gain_meters: row.elevation_gain_meters || exists.elevation_gain_meters,
elevation_loss_meters: row.elevation_loss_meters || exists.elevation_loss_meters,
min_elevation_meters: row.min_elevation_meters || exists.min_elevation_meters,
max_elevation_meters: row.max_elevation_meters || exists.max_elevation_meters,
file_distance_meters: row.file_distance_meters || exists.file_distance_meters,
file_elevation_gain_meters: row.file_elevation_gain_meters || exists.file_elevation_gain_meters,
computed_distance_meters: row.computed_distance_meters || exists.computed_distance_meters,
computed_elevation_gain_meters: row.computed_elevation_gain_meters || exists.computed_elevation_gain_meters,
average_speed_mps: row.average_speed_mps || exists.average_speed_mps,
max_speed_mps: row.max_speed_mps || exists.max_speed_mps,
average_pace_sec_per_km: row.average_pace_sec_per_km || exists.average_pace_sec_per_km,
average_heart_rate: row.average_heart_rate || exists.average_heart_rate,
max_heart_rate: row.max_heart_rate || exists.max_heart_rate,
average_cadence: row.average_cadence || exists.average_cadence,
max_cadence: row.max_cadence || exists.max_cadence,
calories: row.calories || exists.calories,
power_avg: row.power_avg || exists.power_avg,
power_max: row.power_max || exists.power_max,
point_count: row.point_count || exists.point_count,
segment_count: row.segment_count || exists.segment_count,
lap_count: row.lap_count || exists.lap_count,
min_latitude: row.min_latitude || exists.min_latitude,
max_latitude: row.max_latitude || exists.max_latitude,
min_longitude: row.min_longitude || exists.min_longitude,
max_longitude: row.max_longitude || exists.max_longitude,
center_latitude: row.center_latitude || exists.center_latitude,
center_longitude: row.center_longitude || exists.center_longitude,
size_bytes: row.size_bytes || exists.size_bytes,
last_modified_millis: row.last_modified_millis || exists.last_modified_millis,
date_added_millis: row.date_added_millis || exists.date_added_millis,
date_modified_millis: row.date_modified_millis || exists.date_modified_millis,
last_scan_millis: row.last_scan_millis || exists.last_scan_millis,
is_favorite: row.is_favorite ?? exists.is_favorite,
is_hidden: row.is_hidden ?? exists.is_hidden,
rating: row.rating ?? exists.rating,
tags: row.tags ? JSON.stringify(row.tags) : exists.tags,
scan_status: row.scan_status || exists.scan_status,
scan_error: row.scan_error || exists.scan_error,
diagnostics: row.diagnostics ? JSON.stringify(row.diagnostics) : exists.diagnostics,
location: row.location ? JSON.stringify(row.location) : exists.location,
updated_at: knex.fn.now()
});
}
}
/* ============================================================
HARD DELETE
============================================================ */
async function markActivityDeletedHard(id, user) {
await knex('activities').where({ id, user }).del();
await knex('deleted_hard_activities')
.insert({ id, user, deleted_at: knex.fn.now() })
.catch(() => {});
}
/* ============================================================
CHANGES SINCE
============================================================ */
async function getActivitiesSince(user, sinceIso, limit = 1000) {
return knex('activities')
.where({ user })
.andWhere('updated_at', '>', sinceIso)
.orderBy('updated_at', 'asc')
.limit(limit)
.select('*');
}
/* ============================================================
DELETED HARD SINCE
============================================================ */
async function getDeletedHardActivitiesSince(user, sinceIso) {
return knex('deleted_hard_activities')
.where({ user })
.andWhere('deleted_at', '>', sinceIso)
.select('id', 'deleted_at');
}
/* ============================================================
EXPORT
============================================================ */
module.exports = {
insertOrUpdateActivity,
markActivityDeletedHard,
getActivitiesSince,
getDeletedHardActivitiesSince
};

142
db/dbWs.js Normal file
View file

@ -0,0 +1,142 @@
// db/dbWs.js
console.log(">>> dbWs.js CARICATO:", __filename);
const knex = require("./knex");
// ===============================
// SESSIONI
// ===============================
async function upsertSession(session_id, user, device_id) {
const now = Date.now();
const existing = await knex("ws_sessions")
.where({ session_id, device_id })
.first();
if (!existing) {
await knex("ws_sessions").insert({
session_id,
user,
device_id,
connected_at: now,
last_ack: now,
last_sync: null,
need_full_sync: false
});
} else {
await knex("ws_sessions")
.where({ session_id, device_id })
.update({
user,
connected_at: now,
last_ack: now
});
}
}
async function getDevicesForSession(session_id) {
const rows = await knex('ws_sessions')
.where({ session_id })
.select('device_id');
return rows.map(r => r.device_id);
}
async function getSession(session_id, device_id) {
return knex("ws_sessions")
.where({ session_id, device_id })
.first();
}
async function updateSessionAck(session_id, device_id) {
return knex("ws_sessions")
.where({ session_id, device_id })
.update({ last_ack: Date.now() });
}
async function setSessionNeedFullSync(session_id, device_id, value) {
return knex("ws_sessions")
.where({ session_id, device_id })
.update({ need_full_sync: value });
}
async function clearNeedFullSync(session_id, device_id) {
return knex("ws_sessions")
.where({ session_id, device_id })
.update({ need_full_sync: false });
}
async function getSessionsByUser(user) {
return knex("ws_sessions").where({ user });
}
async function deleteSession(session_id, device_id) {
await knex("ws_sessions").where({ session_id, device_id }).del();
await knex("pending_events").where({ session_id, device_id }).del();
}
// ===============================
// PENDING EVENTS (nuovo modello)
// ===============================
async function insertPendingEvent({ user_id, session_id, device_id, event_id, payload, created_at }) {
return knex("pending_events").insert({
user_id,
session_id,
device_id,
event_id,
payload,
created_at
});
}
async function deletePendingEvent({ user_id, session_id, device_id, event_id }) {
return knex("pending_events")
.where({ user_id, session_id, device_id, event_id })
.delete();
}
async function getPendingEvents(user_id, session_id, device_id) {
return knex("pending_events")
.where({ user_id, session_id, device_id })
.orderBy("created_at", "asc");
}
async function deleteAllSessions() {
console.log(">>> deleteAllSessions() – elimino tutte le sessioni WS");
return knex("ws_sessions").del();
}
async function deleteAllPendingEvents() {
console.log(">>> deleteAllPendingEvents() – elimino tutti i pending events");
return knex("pending_events").del();
}
async function setLastSync(session_id, device_id, timestamp = null) {
const ts = timestamp || Date.now();
return knex("ws_sessions")
.where({ session_id, device_id })
.update({ last_sync: ts });
}
// ===============================
// EXPORT
// ===============================
module.exports = {
upsertSession,
getSession,
updateSessionAck,
setSessionNeedFullSync,
clearNeedFullSync,
deleteSession,
getSessionsByUser,
insertPendingEvent,
deletePendingEvent,
getPendingEvents,
getDevicesForSession,
setLastSync,
deleteAllSessions,
deleteAllPendingEvents
};

219
db/faceScanAutomation.js Normal file
View file

@ -0,0 +1,219 @@
async function ensureFaceScanAutomation(db) {
const triggerNames = [
'photos_face_enqueue_insert',
'photos_face_enqueue_update',
'photos_face_cleanup_update',
'photos_face_cleanup_delete',
];
for (const triggerName of triggerNames) {
await db.raw(
`DROP TRIGGER IF EXISTS ${triggerName}`,
);
}
await db.raw(`
CREATE TRIGGER photos_face_enqueue_insert
AFTER INSERT ON photos
WHEN
NEW.deleted_at IS NULL
AND NEW.path IS NOT NULL
AND NEW.path != ''
AND (
NEW.mime_type IS NULL
OR NEW.mime_type LIKE 'image/%'
)
BEGIN
INSERT INTO face_scan_jobs (
photo_id,
status,
attempts,
rescan_requested,
created_at,
updated_at
)
VALUES (
NEW.id,
'pending',
0,
0,
CURRENT_TIMESTAMP,
CURRENT_TIMESTAMP
)
ON CONFLICT(photo_id) DO UPDATE SET
status = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.status
ELSE 'pending'
END,
rescan_requested = CASE
WHEN face_scan_jobs.status = 'processing'
THEN 1
ELSE 0
END,
faces_detected = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.faces_detected
ELSE NULL
END,
execution_ms = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.execution_ms
ELSE NULL
END,
applied_rotation = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.applied_rotation
ELSE NULL
END,
last_error = NULL,
started_at = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.started_at
ELSE NULL
END,
completed_at = NULL,
updated_at = CURRENT_TIMESTAMP;
END
`);
await db.raw(`
CREATE TRIGGER photos_face_enqueue_update
AFTER UPDATE OF
path,
fast_hash,
mtimeMs,
size_bytes,
rotation,
mime_type,
deleted_at
ON photos
WHEN
NEW.deleted_at IS NULL
AND NEW.path IS NOT NULL
AND NEW.path != ''
AND (
NEW.mime_type IS NULL
OR NEW.mime_type LIKE 'image/%'
)
AND (
OLD.path IS NOT NEW.path
OR OLD.fast_hash IS NOT NEW.fast_hash
OR OLD.mtimeMs IS NOT NEW.mtimeMs
OR OLD.size_bytes IS NOT NEW.size_bytes
OR OLD.rotation IS NOT NEW.rotation
OR OLD.mime_type IS NOT NEW.mime_type
OR OLD.deleted_at IS NOT NEW.deleted_at
)
BEGIN
INSERT INTO face_scan_jobs (
photo_id,
status,
attempts,
rescan_requested,
created_at,
updated_at
)
VALUES (
NEW.id,
'pending',
0,
0,
CURRENT_TIMESTAMP,
CURRENT_TIMESTAMP
)
ON CONFLICT(photo_id) DO UPDATE SET
status = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.status
ELSE 'pending'
END,
rescan_requested = CASE
WHEN face_scan_jobs.status = 'processing'
THEN 1
ELSE 0
END,
faces_detected = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.faces_detected
ELSE NULL
END,
execution_ms = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.execution_ms
ELSE NULL
END,
applied_rotation = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.applied_rotation
ELSE NULL
END,
last_error = NULL,
started_at = CASE
WHEN face_scan_jobs.status = 'processing'
THEN face_scan_jobs.started_at
ELSE NULL
END,
completed_at = NULL,
updated_at = CURRENT_TIMESTAMP;
END
`);
await db.raw(`
CREATE TRIGGER photos_face_cleanup_update
AFTER UPDATE OF
path,
mime_type,
deleted_at
ON photos
WHEN
NEW.deleted_at IS NOT NULL
OR NEW.path IS NULL
OR NEW.path = ''
OR (
NEW.mime_type IS NOT NULL
AND NEW.mime_type NOT LIKE 'image/%'
)
BEGIN
DELETE FROM faces
WHERE photo_id = NEW.id;
DELETE FROM face_scan_jobs
WHERE photo_id = NEW.id;
END
`);
await db.raw(`
CREATE TRIGGER photos_face_cleanup_delete
AFTER DELETE ON photos
BEGIN
DELETE FROM faces
WHERE photo_id = OLD.id;
DELETE FROM face_scan_jobs
WHERE photo_id = OLD.id;
END
`);
console.log(
'✔ trigger automatici scansione volti creati',
);
}
module.exports = {
ensureFaceScanAutomation,
};

547
db/init.j Normal file
View file

@ -0,0 +1,547 @@
// db/init.js
const db = require('./knex');
/* ============================================================
FOTO
============================================================ */
async function ensurePhotos() {
const exists = await db.schema.hasTable('photos');
if (!exists) {
await db.schema.createTable('photos', (t) => {
t.string('id').primary();
t.string('user');
t.string('cartella');
t.string('name');
t.string('path');
t.text('extPath').notNullable().defaultTo('');
t.text('extMedia').notNullable().defaultTo('');
t.integer('hybrid').notNullable().defaultTo(0);
t.string('thub1');
t.string('thub2');
t.string('cast');
t.string('mime_type');
t.integer('width');
t.integer('height');
t.integer('rotation');
t.integer('size_bytes');
t.integer('mtimeMs');
t.integer('duration_ms');
t.string('taken_at');
t.string('data');
t.float('lat');
t.float('lon');
t.float('alt');
t.text('location');
t.string('_indexHash');
t.string('fast_hash');
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.datetime('deleted_at').nullable();
});
console.log("✔ photos creata");
} else {
const cols = [
'thub1','thub2','fast_hash','created_at','updated_at','deleted_at'
];
for (const col of cols) {
const existsCol = await db.schema.hasColumn('photos', col);
if (!existsCol) {
await db.schema.alterTable('photos', (t) => {
if (col === 'created_at' || col === 'updated_at')
t.datetime(col).defaultTo(db.fn.now());
else if (col === 'deleted_at')
t.datetime(col).nullable();
else
t.string(col);
});
console.log(`✔ aggiunta colonna photos.${col}`);
}
}
}
}
/* ============================================================
FOTO — deleted_hard
============================================================ */
async function ensureDeletedHard() {
const exists = await db.schema.hasTable('deleted_hard');
if (!exists) {
await db.schema.createTable('deleted_hard', (t) => {
t.string('id').primary();
t.string('user');
t.datetime('deleted_at').defaultTo(db.fn.now());
});
console.log("✔ deleted_hard creata");
}
}
/* ============================================================
WS
============================================================ */
async function ensureWsSessions() {
const exists = await db.schema.hasTable('ws_sessions');
if (!exists) {
await db.schema.createTable('ws_sessions', (t) => {
t.string('session_id').notNullable();
t.string('device_id').notNullable();
t.string('user').notNullable();
t.integer('connected_at').notNullable();
t.integer('last_ack').notNullable();
t.integer('last_sync').nullable();
t.boolean('need_full_sync').notNullable().defaultTo(false);
t.primary(['session_id', 'device_id']);
});
console.log("✔ ws_sessions creata");
} else {
const cols = [
{ name: 'device_id', type: 'string' },
{ name: 'last_sync', type: 'integer' }
];
for (const c of cols) {
const existsCol = await db.schema.hasColumn('ws_sessions', c.name);
if (!existsCol) {
await db.schema.alterTable('ws_sessions', (t) => {
if (c.type === 'string') t.string(c.name).notNullable().defaultTo('default');
if (c.type === 'integer') t.integer(c.name).nullable();
});
console.log(`✔ aggiunta colonna ws_sessions.${c.name}`);
}
}
}
}
async function ensureWsPending() {
const exists = await db.schema.hasTable('ws_pending_events');
if (!exists) {
await db.schema.createTable('ws_pending_events', (t) => {
t.string('event_id').primary();
t.string('session_id').notNullable();
t.text('payload').notNullable();
t.integer('sent_at').notNullable();
t.integer('retries').notNullable().defaultTo(0);
});
console.log("✔ ws_pending_events creata");
}
}
async function ensurePendingEvents() {
const exists = await db.schema.hasTable('pending_events');
if (!exists) {
await db.schema.createTable('pending_events', (t) => {
t.increments('id').primary();
t.string('user_id').notNullable();
t.string('session_id').notNullable();
t.string('device_id').notNullable();
t.string('event_id').notNullable();
t.text('payload').notNullable();
t.integer('created_at').notNullable();
t.unique(['user_id', 'session_id', 'device_id', 'event_id']);
});
console.log("✔ pending_events creata");
}
}
/* ============================================================
🏃 ATTIVITÀ — nuova tabella
============================================================ */
async function ensureActivities() {
const exists = await db.schema.hasTable('activities');
if (!exists) {
await db.schema.createTable('activities', (t) => {
t.string('id').primary();
t.string('user');
t.string('cartella');
t.string('file_name');
t.string('file_extension');
t.string('format');
t.string('display_name');
t.string('route_name');
t.string('title');
t.string('description');
t.string('notes');
t.string('sport');
t.string('activity_type');
t.string('device_manufacturer');
t.string('device_model');
t.string('source_app');
t.bigInteger('start_time_millis');
t.bigInteger('end_time_millis');
t.bigInteger('duration_millis');
t.float('distance_meters');
t.float('elevation_gain_meters');
t.float('elevation_loss_meters');
t.float('min_elevation_meters');
t.float('max_elevation_meters');
t.float('file_distance_meters');
t.float('file_elevation_gain_meters');
t.float('computed_distance_meters');
t.float('computed_elevation_gain_meters');
t.float('average_speed_mps');
t.float('max_speed_mps');
t.float('average_pace_sec_per_km');
t.integer('average_heart_rate');
t.integer('max_heart_rate');
t.integer('average_cadence');
t.integer('max_cadence');
t.integer('calories');
t.integer('power_avg');
t.integer('power_max');
t.integer('point_count');
t.integer('segment_count');
t.integer('lap_count');
t.float('min_latitude');
t.float('max_latitude');
t.float('min_longitude');
t.float('max_longitude');
t.float('center_latitude');
t.float('center_longitude');
t.bigInteger('size_bytes');
t.bigInteger('last_modified_millis');
t.bigInteger('date_added_millis');
t.bigInteger('date_modified_millis');
t.bigInteger('last_scan_millis');
t.boolean('is_favorite');
t.boolean('is_hidden');
t.integer('rating');
t.text('tags');
t.text('track_preview');
t.string('scan_status');
t.text('scan_error');
t.string('path');
t.text('extPath').notNullable().defaultTo('');
t.integer('hybrid').notNullable().defaultTo(0);
t.text('diagnostics');
t.text('location');
t.string('fast_hash');
t.datetime('deleted_at').nullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
});
console.log("✔ activities creata");
}
}
/* ============================================================
ATTIVITÀ — deleted_hard
============================================================ */
async function ensureDeletedHardActivities() {
const exists = await db.schema.hasTable('deleted_hard_activities');
if (!exists) {
await db.schema.createTable('deleted_hard_activities', (t) => {
t.string('id').primary();
t.string('user');
t.datetime('deleted_at').defaultTo(db.fn.now());
});
console.log("✔ deleted_hard_activities creata");
}
}
/* ============================================================
ATTIVITÀ — sync_done
============================================================ */
async function ensureSyncDoneActivities() {
const exists = await db.schema.hasTable('sync_done_activities');
if (!exists) {
await db.schema.createTable('sync_done_activities', (t) => {
t.string('session_id');
t.string('device_id');
t.bigInteger('last_sync');
});
console.log("✔ sync_done_activities creata");
}
}
/* ============================================================
VOLTI
============================================================ */
async function ensureFaces() {
const exists = await db.schema.hasTable('faces');
if (!exists) {
await db.schema.createTable('faces', (t) => {
// ID univoco del volto
t.string('id').primary();
// Foto da cui è stato estratto
t.string('photo_id').notNullable();
t.integer('face_index').notNullable();
// Bounding box nel frame originale
t.float('x').notNullable();
t.float('y').notNullable();
t.float('width').notNullable();
t.float('height').notNullable();
// Confidence del detector SCRFD
t.float('score').notNullable();
// 5 landmark × 2 coordinate
t.float('landmark_0_x');
t.float('landmark_0_y');
t.float('landmark_1_x');
t.float('landmark_1_y');
t.float('landmark_2_x');
t.float('landmark_2_y');
t.float('landmark_3_x');
t.float('landmark_3_y');
t.float('landmark_4_x');
t.float('landmark_4_y');
// 128 × float32 = 512 byte
t.binary('embedding').notNullable();
// Risultato HDBSCAN.
// NULL = non ancora clusterizzato.
t.string('cluster_id').nullable();
// Persona assegnata dall'utente.
// NULL = identità non ancora assegnata.
t.string('person_id').nullable();
// Stato applicativo del volto.
// detected / clustered / assigned
t.string('status').notNullable().defaultTo('detected');
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
// Indici principali
t.index(['photo_id']);
t.index(['cluster_id']);
t.index(['person_id']);
t.index(['status']);
t.unique(['photo_id', 'face_index']);
// Evita due record per lo stesso volto nella stessa foto
// solo se il nostro scanner assegnerà un face_index.
});
console.log("✔ faces creata");
}
}
/* ============================================================
CODA SCANSIONE VOLTI
============================================================ */
async function ensureFaceScanJobs() {
const exists =
await db.schema.hasTable('face_scan_jobs');
if (!exists) {
await db.schema.createTable(
'face_scan_jobs',
(t) => {
t.string('photo_id').primary();
// pending / processing / done / error
t.string('status')
.notNullable()
.defaultTo('pending');
t.integer('attempts')
.notNullable()
.defaultTo(0);
t.integer('rescan_requested')
.notNullable()
.defaultTo(0);
t.integer('faces_detected').nullable();
t.integer('execution_ms').nullable();
t.integer('applied_rotation').nullable();
t.text('last_error').nullable();
t.datetime('started_at').nullable();
t.datetime('completed_at').nullable();
t.datetime('created_at')
.defaultTo(db.fn.now());
t.datetime('updated_at')
.defaultTo(db.fn.now());
t.index(['status']);
t.index(['updated_at']);
},
);
console.log("✔ face_scan_jobs creata");
} else {
const hasRescanRequested =
await db.schema.hasColumn(
'face_scan_jobs',
'rescan_requested',
);
if (!hasRescanRequested) {
await db.schema.alterTable(
'face_scan_jobs',
(t) => {
t.integer('rescan_requested')
.notNullable()
.defaultTo(0);
},
);
console.log(
"✔ aggiunta face_scan_jobs.rescan_requested",
);
}
}
}
/* ============================================================
CLUSTER HDBSCAN
============================================================ */
async function ensureFaceClusters() {
const exists = await db.schema.hasTable('face_clusters');
if (!exists) {
await db.schema.createTable('face_clusters', (t) => {
// ID persistente del cluster
t.string('id').primary();
// Identifica una specifica esecuzione di HDBSCAN.
// Esempio: "initial-2026-09-23T04:20:00"
t.string('run_id').notNullable();
// Label numerica prodotta da HDBSCAN.
// Può essere -1 per noise.
t.integer('label').notNullable();
// Numero di facce nel cluster
t.integer('size').notNullable().defaultTo(0);
// Parametri principali usati per generare il cluster
t.integer('min_samples');
t.integer('min_cluster_size');
// NULL se il cluster non è ancora stato
// associato ad una persona.
t.string('person_id').nullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.index(['run_id']);
t.index(['person_id']);
// Un label può comparire una sola volta
// all'interno dello stesso run.
t.unique(['run_id', 'label']);
});
console.log("✔ face_clusters creata");
}
}
/* ============================================================
PERSONE
============================================================ */
async function ensurePersons() {
const exists = await db.schema.hasTable('persons');
if (!exists) {
await db.schema.createTable('persons', (t) => {
// ID stabile dell'identità applicativa
t.string('id').primary();
// Utente proprietario della libreria
t.string('user').nullable();
// Nome assegnato dall'utente
t.string('name').notNullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.index(['user']);
t.index(['name']);
});
console.log("✔ persons creata");
}
}
/* ============================================================
INIT
============================================================ */
/* async function init() {
try {
console.log("🔧 init DB...");
await ensurePhotos();
await ensureDeletedHard();
await ensureActivities();
await ensureDeletedHardActivities();
await ensureSyncDoneActivities();
await ensureWsSessions();
await ensureWsPending();
await ensurePendingEvents();
console.log("✅ init completato");
process.exit(0);
} catch (err) {
console.error("❌ errore init:", err);
process.exit(1);
}
}
*/
async function init() {
try {
console.log("🔧 init DB...");
await ensurePhotos();
await ensureDeletedHard();
await ensureActivities();
await ensureDeletedHardActivities();
await ensureSyncDoneActivities();
await ensureWsSessions();
await ensureWsPending();
await ensurePendingEvents();
// Face recognition
await ensureFaces();
await ensureFaceScanJobs();
await ensureFaceClusters();
await ensurePersons();
console.log("✅ init completato");
process.exit(0);
} catch (err) {
console.error("❌ errore init:", err);
process.exit(1);
}
}
init();

1321
db/init.js Normal file

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,521 @@
// db/init.js
const db = require('./knex');
/* ============================================================
FOTO
============================================================ */
async function ensurePhotos() {
const exists = await db.schema.hasTable('photos');
if (!exists) {
await db.schema.createTable('photos', (t) => {
t.string('id').primary();
t.string('user');
t.string('cartella');
t.string('name');
t.string('path');
t.text('extPath').notNullable().defaultTo('');
t.text('extMedia').notNullable().defaultTo('');
t.integer('hybrid').notNullable().defaultTo(0);
t.string('thub1');
t.string('thub2');
t.string('cast');
t.string('mime_type');
t.integer('width');
t.integer('height');
t.integer('rotation');
t.integer('size_bytes');
t.integer('mtimeMs');
t.integer('duration_ms');
t.string('taken_at');
t.string('data');
t.float('lat');
t.float('lon');
t.float('alt');
t.text('location');
t.string('_indexHash');
t.string('fast_hash');
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.datetime('deleted_at').nullable();
});
console.log("✔ photos creata");
} else {
const cols = [
'thub1','thub2','fast_hash','created_at','updated_at','deleted_at'
];
for (const col of cols) {
const existsCol = await db.schema.hasColumn('photos', col);
if (!existsCol) {
await db.schema.alterTable('photos', (t) => {
if (col === 'created_at' || col === 'updated_at')
t.datetime(col).defaultTo(db.fn.now());
else if (col === 'deleted_at')
t.datetime(col).nullable();
else
t.string(col);
});
console.log(`✔ aggiunta colonna photos.${col}`);
}
}
}
}
/* ============================================================
FOTO — deleted_hard
============================================================ */
async function ensureDeletedHard() {
const exists = await db.schema.hasTable('deleted_hard');
if (!exists) {
await db.schema.createTable('deleted_hard', (t) => {
t.string('id').primary();
t.string('user');
t.datetime('deleted_at').defaultTo(db.fn.now());
});
console.log("✔ deleted_hard creata");
}
}
/* ============================================================
WS
============================================================ */
async function ensureWsSessions() {
const exists = await db.schema.hasTable('ws_sessions');
if (!exists) {
await db.schema.createTable('ws_sessions', (t) => {
t.string('session_id').notNullable();
t.string('device_id').notNullable();
t.string('user').notNullable();
t.integer('connected_at').notNullable();
t.integer('last_ack').notNullable();
t.integer('last_sync').nullable();
t.boolean('need_full_sync').notNullable().defaultTo(false);
t.primary(['session_id', 'device_id']);
});
console.log("✔ ws_sessions creata");
} else {
const cols = [
{ name: 'device_id', type: 'string' },
{ name: 'last_sync', type: 'integer' }
];
for (const c of cols) {
const existsCol = await db.schema.hasColumn('ws_sessions', c.name);
if (!existsCol) {
await db.schema.alterTable('ws_sessions', (t) => {
if (c.type === 'string') t.string(c.name).notNullable().defaultTo('default');
if (c.type === 'integer') t.integer(c.name).nullable();
});
console.log(`✔ aggiunta colonna ws_sessions.${c.name}`);
}
}
}
}
async function ensureWsPending() {
const exists = await db.schema.hasTable('ws_pending_events');
if (!exists) {
await db.schema.createTable('ws_pending_events', (t) => {
t.string('event_id').primary();
t.string('session_id').notNullable();
t.text('payload').notNullable();
t.integer('sent_at').notNullable();
t.integer('retries').notNullable().defaultTo(0);
});
console.log("✔ ws_pending_events creata");
}
}
async function ensurePendingEvents() {
const exists = await db.schema.hasTable('pending_events');
if (!exists) {
await db.schema.createTable('pending_events', (t) => {
t.increments('id').primary();
t.string('user_id').notNullable();
t.string('session_id').notNullable();
t.string('device_id').notNullable();
t.string('event_id').notNullable();
t.text('payload').notNullable();
t.integer('created_at').notNullable();
t.unique(['user_id', 'session_id', 'device_id', 'event_id']);
});
console.log("✔ pending_events creata");
}
}
/* ============================================================
🏃 ATTIVITÀ — nuova tabella
============================================================ */
async function ensureActivities() {
const exists = await db.schema.hasTable('activities');
if (!exists) {
await db.schema.createTable('activities', (t) => {
t.string('id').primary();
t.string('user');
t.string('cartella');
t.string('file_name');
t.string('file_extension');
t.string('format');
t.string('display_name');
t.string('route_name');
t.string('title');
t.string('description');
t.string('notes');
t.string('sport');
t.string('activity_type');
t.string('device_manufacturer');
t.string('device_model');
t.string('source_app');
t.bigInteger('start_time_millis');
t.bigInteger('end_time_millis');
t.bigInteger('duration_millis');
t.float('distance_meters');
t.float('elevation_gain_meters');
t.float('elevation_loss_meters');
t.float('min_elevation_meters');
t.float('max_elevation_meters');
t.float('file_distance_meters');
t.float('file_elevation_gain_meters');
t.float('computed_distance_meters');
t.float('computed_elevation_gain_meters');
t.float('average_speed_mps');
t.float('max_speed_mps');
t.float('average_pace_sec_per_km');
t.integer('average_heart_rate');
t.integer('max_heart_rate');
t.integer('average_cadence');
t.integer('max_cadence');
t.integer('calories');
t.integer('power_avg');
t.integer('power_max');
t.integer('point_count');
t.integer('segment_count');
t.integer('lap_count');
t.float('min_latitude');
t.float('max_latitude');
t.float('min_longitude');
t.float('max_longitude');
t.float('center_latitude');
t.float('center_longitude');
t.bigInteger('size_bytes');
t.bigInteger('last_modified_millis');
t.bigInteger('date_added_millis');
t.bigInteger('date_modified_millis');
t.bigInteger('last_scan_millis');
t.boolean('is_favorite');
t.boolean('is_hidden');
t.integer('rating');
t.text('tags');
t.text('track_preview');
t.string('scan_status');
t.text('scan_error');
t.string('path');
t.text('extPath').notNullable().defaultTo('');
t.integer('hybrid').notNullable().defaultTo(0);
t.text('diagnostics');
t.text('location');
t.string('fast_hash');
t.datetime('deleted_at').nullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
});
console.log("✔ activities creata");
}
}
/* ============================================================
ATTIVITÀ — deleted_hard
============================================================ */
async function ensureDeletedHardActivities() {
const exists = await db.schema.hasTable('deleted_hard_activities');
if (!exists) {
await db.schema.createTable('deleted_hard_activities', (t) => {
t.string('id').primary();
t.string('user');
t.datetime('deleted_at').defaultTo(db.fn.now());
});
console.log("✔ deleted_hard_activities creata");
}
}
/* ============================================================
ATTIVITÀ — sync_done
============================================================ */
async function ensureSyncDoneActivities() {
const exists = await db.schema.hasTable('sync_done_activities');
if (!exists) {
await db.schema.createTable('sync_done_activities', (t) => {
t.string('session_id');
t.string('device_id');
t.bigInteger('last_sync');
});
console.log("✔ sync_done_activities creata");
}
}
/* ============================================================
VOLTI
============================================================ */
async function ensureFaces() {
const exists = await db.schema.hasTable('faces');
if (!exists) {
await db.schema.createTable('faces', (t) => {
// ID univoco del volto
t.string('id').primary();
// Foto da cui è stato estratto
t.string('photo_id').notNullable();
t.integer('face_index').notNullable();
// Bounding box nel frame originale
t.float('x').notNullable();
t.float('y').notNullable();
t.float('width').notNullable();
t.float('height').notNullable();
// Confidence del detector SCRFD
t.float('score').notNullable();
// 5 landmark × 2 coordinate
t.float('landmark_0_x');
t.float('landmark_0_y');
t.float('landmark_1_x');
t.float('landmark_1_y');
t.float('landmark_2_x');
t.float('landmark_2_y');
t.float('landmark_3_x');
t.float('landmark_3_y');
t.float('landmark_4_x');
t.float('landmark_4_y');
// 128 × float32 = 512 byte
t.binary('embedding').notNullable();
// Risultato HDBSCAN.
// NULL = non ancora clusterizzato.
t.string('cluster_id').nullable();
// Persona assegnata dall'utente.
// NULL = identità non ancora assegnata.
t.string('person_id').nullable();
// Stato applicativo del volto.
// detected / clustered / assigned
t.string('status').notNullable().defaultTo('detected');
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
// Indici principali
t.index(['photo_id']);
t.index(['cluster_id']);
t.index(['person_id']);
t.index(['status']);
t.unique(['photo_id', 'face_index']);
// Evita due record per lo stesso volto nella stessa foto
// solo se il nostro scanner assegnerà un face_index.
});
console.log("✔ faces creata");
}
}
/* ============================================================
CODA SCANSIONE VOLTI
============================================================ */
async function ensureFaceScanJobs() {
const exists =
await db.schema.hasTable('face_scan_jobs');
if (!exists) {
await db.schema.createTable(
'face_scan_jobs',
(t) => {
t.string('photo_id').primary();
// pending / processing / done / error
t.string('status')
.notNullable()
.defaultTo('pending');
t.integer('attempts')
.notNullable()
.defaultTo(0);
t.integer('faces_detected').nullable();
t.integer('execution_ms').nullable();
t.integer('applied_rotation').nullable();
t.text('last_error').nullable();
t.datetime('started_at').nullable();
t.datetime('completed_at').nullable();
t.datetime('created_at')
.defaultTo(db.fn.now());
t.datetime('updated_at')
.defaultTo(db.fn.now());
t.index(['status']);
t.index(['updated_at']);
},
);
console.log("✔ face_scan_jobs creata");
}
}
/* ============================================================
CLUSTER HDBSCAN
============================================================ */
async function ensureFaceClusters() {
const exists = await db.schema.hasTable('face_clusters');
if (!exists) {
await db.schema.createTable('face_clusters', (t) => {
// ID persistente del cluster
t.string('id').primary();
// Identifica una specifica esecuzione di HDBSCAN.
// Esempio: "initial-2026-09-23T04:20:00"
t.string('run_id').notNullable();
// Label numerica prodotta da HDBSCAN.
// Può essere -1 per noise.
t.integer('label').notNullable();
// Numero di facce nel cluster
t.integer('size').notNullable().defaultTo(0);
// Parametri principali usati per generare il cluster
t.integer('min_samples');
t.integer('min_cluster_size');
// NULL se il cluster non è ancora stato
// associato ad una persona.
t.string('person_id').nullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.index(['run_id']);
t.index(['person_id']);
// Un label può comparire una sola volta
// all'interno dello stesso run.
t.unique(['run_id', 'label']);
});
console.log("✔ face_clusters creata");
}
}
/* ============================================================
PERSONE
============================================================ */
async function ensurePersons() {
const exists = await db.schema.hasTable('persons');
if (!exists) {
await db.schema.createTable('persons', (t) => {
// ID stabile dell'identità applicativa
t.string('id').primary();
// Utente proprietario della libreria
t.string('user').nullable();
// Nome assegnato dall'utente
t.string('name').notNullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.index(['user']);
t.index(['name']);
});
console.log("✔ persons creata");
}
}
/* ============================================================
INIT
============================================================ */
/* async function init() {
try {
console.log("🔧 init DB...");
await ensurePhotos();
await ensureDeletedHard();
await ensureActivities();
await ensureDeletedHardActivities();
await ensureSyncDoneActivities();
await ensureWsSessions();
await ensureWsPending();
await ensurePendingEvents();
console.log("✅ init completato");
process.exit(0);
} catch (err) {
console.error("❌ errore init:", err);
process.exit(1);
}
}
*/
async function init() {
try {
console.log("🔧 init DB...");
await ensurePhotos();
await ensureDeletedHard();
await ensureActivities();
await ensureDeletedHardActivities();
await ensureSyncDoneActivities();
await ensureWsSessions();
await ensureWsPending();
await ensurePendingEvents();
// Face recognition
await ensureFaces();
await ensureFaceScanJobs();
await ensureFaceClusters();
await ensurePersons();
console.log("✅ init completato");
process.exit(0);
} catch (err) {
console.error("❌ errore init:", err);
process.exit(1);
}
}
init();

View file

@ -0,0 +1,783 @@
// db/init.js
const db = require('./knex');
const {
ensureFaceScanAutomation,
} = require('./faceScanAutomation');
/* ============================================================
FOTO
============================================================ */
async function ensurePhotos() {
const exists = await db.schema.hasTable('photos');
if (!exists) {
await db.schema.createTable('photos', (t) => {
t.string('id').primary();
t.string('user');
t.string('cartella');
t.string('name');
t.string('path');
t.text('extPath').notNullable().defaultTo('');
t.text('extMedia').notNullable().defaultTo('');
t.integer('hybrid').notNullable().defaultTo(0);
t.string('thub1');
t.string('thub2');
t.string('cast');
t.string('mime_type');
t.integer('width');
t.integer('height');
t.integer('rotation');
t.integer('size_bytes');
t.integer('mtimeMs');
t.integer('duration_ms');
t.string('taken_at');
t.string('data');
t.float('lat');
t.float('lon');
t.float('alt');
t.text('location');
t.string('_indexHash');
t.string('fast_hash');
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.datetime('deleted_at').nullable();
});
console.log("✔ photos creata");
} else {
const cols = [
'thub1','thub2','fast_hash','created_at','updated_at','deleted_at'
];
for (const col of cols) {
const existsCol = await db.schema.hasColumn('photos', col);
if (!existsCol) {
await db.schema.alterTable('photos', (t) => {
if (col === 'created_at' || col === 'updated_at')
t.datetime(col).defaultTo(db.fn.now());
else if (col === 'deleted_at')
t.datetime(col).nullable();
else
t.string(col);
});
console.log(`✔ aggiunta colonna photos.${col}`);
}
}
}
}
/* ============================================================
FOTO — deleted_hard
============================================================ */
async function ensureDeletedHard() {
const exists = await db.schema.hasTable('deleted_hard');
if (!exists) {
await db.schema.createTable('deleted_hard', (t) => {
t.string('id').primary();
t.string('user');
t.datetime('deleted_at').defaultTo(db.fn.now());
});
console.log("✔ deleted_hard creata");
}
}
/* ============================================================
WS
============================================================ */
async function ensureWsSessions() {
const exists = await db.schema.hasTable('ws_sessions');
if (!exists) {
await db.schema.createTable('ws_sessions', (t) => {
t.string('session_id').notNullable();
t.string('device_id').notNullable();
t.string('user').notNullable();
t.integer('connected_at').notNullable();
t.integer('last_ack').notNullable();
t.integer('last_sync').nullable();
t.boolean('need_full_sync').notNullable().defaultTo(false);
t.primary(['session_id', 'device_id']);
});
console.log("✔ ws_sessions creata");
} else {
const cols = [
{ name: 'device_id', type: 'string' },
{ name: 'last_sync', type: 'integer' }
];
for (const c of cols) {
const existsCol = await db.schema.hasColumn('ws_sessions', c.name);
if (!existsCol) {
await db.schema.alterTable('ws_sessions', (t) => {
if (c.type === 'string') t.string(c.name).notNullable().defaultTo('default');
if (c.type === 'integer') t.integer(c.name).nullable();
});
console.log(`✔ aggiunta colonna ws_sessions.${c.name}`);
}
}
}
}
async function ensureWsPending() {
const exists = await db.schema.hasTable('ws_pending_events');
if (!exists) {
await db.schema.createTable('ws_pending_events', (t) => {
t.string('event_id').primary();
t.string('session_id').notNullable();
t.text('payload').notNullable();
t.integer('sent_at').notNullable();
t.integer('retries').notNullable().defaultTo(0);
});
console.log("✔ ws_pending_events creata");
}
}
async function ensurePendingEvents() {
const exists = await db.schema.hasTable('pending_events');
if (!exists) {
await db.schema.createTable('pending_events', (t) => {
t.increments('id').primary();
t.string('user_id').notNullable();
t.string('session_id').notNullable();
t.string('device_id').notNullable();
t.string('event_id').notNullable();
t.text('payload').notNullable();
t.integer('created_at').notNullable();
t.unique(['user_id', 'session_id', 'device_id', 'event_id']);
});
console.log("✔ pending_events creata");
}
}
/* ============================================================
🏃 ATTIVITÀ — nuova tabella
============================================================ */
async function ensureActivities() {
const exists = await db.schema.hasTable('activities');
if (!exists) {
await db.schema.createTable('activities', (t) => {
t.string('id').primary();
t.string('user');
t.string('cartella');
t.string('file_name');
t.string('file_extension');
t.string('format');
t.string('display_name');
t.string('route_name');
t.string('title');
t.string('description');
t.string('notes');
t.string('sport');
t.string('activity_type');
t.string('device_manufacturer');
t.string('device_model');
t.string('source_app');
t.bigInteger('start_time_millis');
t.bigInteger('end_time_millis');
t.bigInteger('duration_millis');
t.float('distance_meters');
t.float('elevation_gain_meters');
t.float('elevation_loss_meters');
t.float('min_elevation_meters');
t.float('max_elevation_meters');
t.float('file_distance_meters');
t.float('file_elevation_gain_meters');
t.float('computed_distance_meters');
t.float('computed_elevation_gain_meters');
t.float('average_speed_mps');
t.float('max_speed_mps');
t.float('average_pace_sec_per_km');
t.integer('average_heart_rate');
t.integer('max_heart_rate');
t.integer('average_cadence');
t.integer('max_cadence');
t.integer('calories');
t.integer('power_avg');
t.integer('power_max');
t.integer('point_count');
t.integer('segment_count');
t.integer('lap_count');
t.float('min_latitude');
t.float('max_latitude');
t.float('min_longitude');
t.float('max_longitude');
t.float('center_latitude');
t.float('center_longitude');
t.bigInteger('size_bytes');
t.bigInteger('last_modified_millis');
t.bigInteger('date_added_millis');
t.bigInteger('date_modified_millis');
t.bigInteger('last_scan_millis');
t.boolean('is_favorite');
t.boolean('is_hidden');
t.integer('rating');
t.text('tags');
t.text('track_preview');
t.string('scan_status');
t.text('scan_error');
t.string('path');
t.text('extPath').notNullable().defaultTo('');
t.integer('hybrid').notNullable().defaultTo(0);
t.text('diagnostics');
t.text('location');
t.string('fast_hash');
t.datetime('deleted_at').nullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
});
console.log("✔ activities creata");
}
}
/* ============================================================
ATTIVITÀ — deleted_hard
============================================================ */
async function ensureDeletedHardActivities() {
const exists = await db.schema.hasTable('deleted_hard_activities');
if (!exists) {
await db.schema.createTable('deleted_hard_activities', (t) => {
t.string('id').primary();
t.string('user');
t.datetime('deleted_at').defaultTo(db.fn.now());
});
console.log("✔ deleted_hard_activities creata");
}
}
/* ============================================================
ATTIVITÀ — sync_done
============================================================ */
async function ensureSyncDoneActivities() {
const exists = await db.schema.hasTable('sync_done_activities');
if (!exists) {
await db.schema.createTable('sync_done_activities', (t) => {
t.string('session_id');
t.string('device_id');
t.bigInteger('last_sync');
});
console.log("✔ sync_done_activities creata");
}
}
/* ============================================================
VOLTI
============================================================ */
async function ensureFaces() {
const exists = await db.schema.hasTable('faces');
if (!exists) {
await db.schema.createTable('faces', (t) => {
// ID univoco del volto
t.string('id').primary();
// Foto da cui è stato estratto
t.string('photo_id').notNullable();
t.integer('face_index').notNullable();
// Bounding box nel frame originale
t.float('x').notNullable();
t.float('y').notNullable();
t.float('width').notNullable();
t.float('height').notNullable();
// Confidence del detector SCRFD
t.float('score').notNullable();
// 5 landmark × 2 coordinate
t.float('landmark_0_x');
t.float('landmark_0_y');
t.float('landmark_1_x');
t.float('landmark_1_y');
t.float('landmark_2_x');
t.float('landmark_2_y');
t.float('landmark_3_x');
t.float('landmark_3_y');
t.float('landmark_4_x');
t.float('landmark_4_y');
// 128 × float32 = 512 byte
t.binary('embedding').notNullable();
// Risultato HDBSCAN.
// NULL = non ancora clusterizzato.
t.string('cluster_id').nullable();
// Persona assegnata dall'utente.
// NULL = identità non ancora assegnata.
t.string('person_id').nullable();
// Stato applicativo del volto.
// detected / clustered / assigned
t.string('status').notNullable().defaultTo('detected');
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
// Indici principali
t.index(['photo_id']);
t.index(['cluster_id']);
t.index(['person_id']);
t.index(['status']);
t.unique(['photo_id', 'face_index']);
// Evita due record per lo stesso volto nella stessa foto
// solo se il nostro scanner assegnerà un face_index.
});
console.log("✔ faces creata");
}
}
/* ============================================================
CODA SCANSIONE VOLTI
============================================================ */
async function ensureFaceScanJobs() {
const exists =
await db.schema.hasTable('face_scan_jobs');
if (!exists) {
await db.schema.createTable(
'face_scan_jobs',
(t) => {
t.string('photo_id').primary();
// pending / processing / done / error
t.string('status')
.notNullable()
.defaultTo('pending');
t.integer('attempts')
.notNullable()
.defaultTo(0);
t.integer('rescan_requested')
.notNullable()
.defaultTo(0);
t.integer('faces_detected').nullable();
t.integer('execution_ms').nullable();
t.integer('applied_rotation').nullable();
t.text('last_error').nullable();
t.datetime('started_at').nullable();
t.datetime('completed_at').nullable();
t.datetime('created_at')
.defaultTo(db.fn.now());
t.datetime('updated_at')
.defaultTo(db.fn.now());
t.index(['status']);
t.index(['updated_at']);
},
);
console.log("✔ face_scan_jobs creata");
} else {
const hasRescanRequested =
await db.schema.hasColumn(
'face_scan_jobs',
'rescan_requested',
);
if (!hasRescanRequested) {
await db.schema.alterTable(
'face_scan_jobs',
(t) => {
t.integer('rescan_requested')
.notNullable()
.defaultTo(0);
},
);
console.log(
"✔ aggiunta face_scan_jobs.rescan_requested",
);
}
}
}
/* ============================================================
CLUSTER HDBSCAN
============================================================ */
async function ensureFaceClusters() {
const exists = await db.schema.hasTable('face_clusters');
if (!exists) {
await db.schema.createTable('face_clusters', (t) => {
// ID persistente del cluster
t.string('id').primary();
// Identifica una specifica esecuzione di HDBSCAN.
// Esempio: "initial-2026-09-23T04:20:00"
t.string('run_id').notNullable();
// Label numerica prodotta da HDBSCAN.
// Può essere -1 per noise.
t.integer('label').notNullable();
// Numero di facce nel cluster
t.integer('size').notNullable().defaultTo(0);
// Parametri principali usati per generare il cluster
t.integer('min_samples');
t.integer('min_cluster_size');
// NULL se il cluster non è ancora stato
// associato ad una persona.
t.string('person_id').nullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.index(['run_id']);
t.index(['person_id']);
// Un label può comparire una sola volta
// all'interno dello stesso run.
t.unique(['run_id', 'label']);
});
console.log("✔ face_clusters creata");
}
}
/* ============================================================
RUN CLUSTERING VOLTI
============================================================ */
async function ensureFaceClusterRuns() {
const exists =
await db.schema.hasTable(
'face_cluster_runs',
);
if (!exists) {
await db.schema.createTable(
'face_cluster_runs',
(t) => {
// Identificativo stabile della run.
t.string('id').primary();
// Proprietario della libreria fotografica.
t.string('user')
.notNullable();
// pending / processing / done / error
t.string('status')
.notNullable()
.defaultTo('pending');
// all / unclustered / unknown
t.string('scope')
.notNullable()
.defaultTo('all');
t.integer('embedding_count')
.notNullable()
.defaultTo(0);
t.integer('min_cluster_size')
.notNullable();
t.integer('min_samples')
.notNullable();
t.string('metric')
.notNullable()
.defaultTo('euclidean');
// eom / leaf
t.string('selection_method')
.notNullable()
.defaultTo('eom');
// sklearn / native / opencl-native
t.string('engine')
.notNullable()
.defaultTo('sklearn');
t.string('engine_version')
.nullable();
t.integer('cluster_count')
.nullable();
t.integer('clustered_faces')
.nullable();
t.integer('noise_faces')
.nullable();
t.float('noise_percent')
.nullable();
t.text('last_error')
.nullable();
t.datetime('started_at')
.nullable();
t.datetime('completed_at')
.nullable();
t.datetime('created_at')
.defaultTo(db.fn.now());
t.datetime('updated_at')
.defaultTo(db.fn.now());
t.index(['user']);
t.index(['status']);
t.index(['user', 'created_at']);
},
);
console.log(
"✔ face_cluster_runs creata",
);
}
}
/* ============================================================
APPARTENENZE DEI VOLTI AI CLUSTER
============================================================ */
async function ensureFaceClusterMemberships() {
const exists =
await db.schema.hasTable(
'face_cluster_memberships',
);
if (!exists) {
await db.schema.createTable(
'face_cluster_memberships',
(t) => {
// Identificativo stabile della membership.
t.string('id').primary();
// Run HDBSCAN che ha prodotto il risultato.
t.string('run_id')
.notNullable();
// NULL per i volti classificati come noise.
t.string('cluster_id')
.nullable();
t.string('face_id')
.notNullable();
// Label originale HDBSCAN.
// -1 indica noise.
t.integer('label')
.notNullable();
// Probabilità di appartenenza HDBSCAN.
t.float('probability')
.notNullable()
.defaultTo(0);
t.integer('noise_flag')
.notNullable()
.defaultTo(0);
// suggested / accepted / excluded
t.string('review_status')
.notNullable()
.defaultTo('suggested');
t.string('reviewed_by')
.nullable();
t.datetime('reviewed_at')
.nullable();
t.datetime('created_at')
.defaultTo(db.fn.now());
t.datetime('updated_at')
.defaultTo(db.fn.now());
// Un volto compare una sola volta per run.
t.unique([
'run_id',
'face_id',
]);
t.index(['run_id']);
t.index(['cluster_id']);
t.index(['face_id']);
t.index(['review_status']);
t.index([
'run_id',
'label',
]);
},
);
console.log(
"✔ face_cluster_memberships creata",
);
}
}
/* ============================================================
ESCLUSIONI MANUALI PERSONA-VOLTO
============================================================ */
async function ensurePersonFaceExclusions() {
const exists =
await db.schema.hasTable(
'person_face_exclusions',
);
if (!exists) {
await db.schema.createTable(
'person_face_exclusions',
(t) => {
// Persona dalla quale il volto è stato escluso.
t.string('person_id')
.notNullable();
t.string('face_id')
.notNullable();
// Proprietario della libreria.
t.string('user')
.notNullable();
t.text('reason')
.nullable();
t.string('created_by')
.nullable();
t.datetime('created_at')
.defaultTo(db.fn.now());
t.primary([
'person_id',
'face_id',
]);
t.index(['user']);
t.index(['face_id']);
},
);
console.log(
"✔ person_face_exclusions creata",
);
}
}
/* ============================================================
PERSONE
============================================================ */
async function ensurePersons() {
const exists = await db.schema.hasTable('persons');
if (!exists) {
await db.schema.createTable('persons', (t) => {
// ID stabile dell'identità applicativa
t.string('id').primary();
// Utente proprietario della libreria
t.string('user').nullable();
// Nome assegnato dall'utente
t.string('name').notNullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.index(['user']);
t.index(['name']);
});
console.log("✔ persons creata");
}
}
/* ============================================================
INIT
============================================================ */
/* async function init() {
try {
console.log("🔧 init DB...");
await ensurePhotos();
await ensureDeletedHard();
await ensureActivities();
await ensureDeletedHardActivities();
await ensureSyncDoneActivities();
await ensureWsSessions();
await ensureWsPending();
await ensurePendingEvents();
console.log("✅ init completato");
process.exit(0);
} catch (err) {
console.error("❌ errore init:", err);
process.exit(1);
}
}
*/
async function init() {
try {
console.log("🔧 init DB...");
await ensurePhotos();
await ensureDeletedHard();
await ensureActivities();
await ensureDeletedHardActivities();
await ensureSyncDoneActivities();
await ensureWsSessions();
await ensureWsPending();
await ensurePendingEvents();
// Face recognition
await ensureFaces();
await ensureFaceScanJobs();
await ensureFaceScanAutomation(db);
await ensureFaceClusters();
await ensureFaceClusterRuns();
await ensureFaceClusterMemberships();
await ensurePersonFaceExclusions();
await ensurePersons();
console.log("✅ init completato");
process.exit(0);
} catch (err) {
console.error("❌ errore init:", err);
process.exit(1);
}
}
init();

View file

@ -0,0 +1,551 @@
// db/init.js
const db = require('./knex');
const {
ensureFaceScanAutomation,
} = require('./faceScanAutomation');
/* ============================================================
FOTO
============================================================ */
async function ensurePhotos() {
const exists = await db.schema.hasTable('photos');
if (!exists) {
await db.schema.createTable('photos', (t) => {
t.string('id').primary();
t.string('user');
t.string('cartella');
t.string('name');
t.string('path');
t.text('extPath').notNullable().defaultTo('');
t.text('extMedia').notNullable().defaultTo('');
t.integer('hybrid').notNullable().defaultTo(0);
t.string('thub1');
t.string('thub2');
t.string('cast');
t.string('mime_type');
t.integer('width');
t.integer('height');
t.integer('rotation');
t.integer('size_bytes');
t.integer('mtimeMs');
t.integer('duration_ms');
t.string('taken_at');
t.string('data');
t.float('lat');
t.float('lon');
t.float('alt');
t.text('location');
t.string('_indexHash');
t.string('fast_hash');
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.datetime('deleted_at').nullable();
});
console.log("✔ photos creata");
} else {
const cols = [
'thub1','thub2','fast_hash','created_at','updated_at','deleted_at'
];
for (const col of cols) {
const existsCol = await db.schema.hasColumn('photos', col);
if (!existsCol) {
await db.schema.alterTable('photos', (t) => {
if (col === 'created_at' || col === 'updated_at')
t.datetime(col).defaultTo(db.fn.now());
else if (col === 'deleted_at')
t.datetime(col).nullable();
else
t.string(col);
});
console.log(`✔ aggiunta colonna photos.${col}`);
}
}
}
}
/* ============================================================
FOTO — deleted_hard
============================================================ */
async function ensureDeletedHard() {
const exists = await db.schema.hasTable('deleted_hard');
if (!exists) {
await db.schema.createTable('deleted_hard', (t) => {
t.string('id').primary();
t.string('user');
t.datetime('deleted_at').defaultTo(db.fn.now());
});
console.log("✔ deleted_hard creata");
}
}
/* ============================================================
WS
============================================================ */
async function ensureWsSessions() {
const exists = await db.schema.hasTable('ws_sessions');
if (!exists) {
await db.schema.createTable('ws_sessions', (t) => {
t.string('session_id').notNullable();
t.string('device_id').notNullable();
t.string('user').notNullable();
t.integer('connected_at').notNullable();
t.integer('last_ack').notNullable();
t.integer('last_sync').nullable();
t.boolean('need_full_sync').notNullable().defaultTo(false);
t.primary(['session_id', 'device_id']);
});
console.log("✔ ws_sessions creata");
} else {
const cols = [
{ name: 'device_id', type: 'string' },
{ name: 'last_sync', type: 'integer' }
];
for (const c of cols) {
const existsCol = await db.schema.hasColumn('ws_sessions', c.name);
if (!existsCol) {
await db.schema.alterTable('ws_sessions', (t) => {
if (c.type === 'string') t.string(c.name).notNullable().defaultTo('default');
if (c.type === 'integer') t.integer(c.name).nullable();
});
console.log(`✔ aggiunta colonna ws_sessions.${c.name}`);
}
}
}
}
async function ensureWsPending() {
const exists = await db.schema.hasTable('ws_pending_events');
if (!exists) {
await db.schema.createTable('ws_pending_events', (t) => {
t.string('event_id').primary();
t.string('session_id').notNullable();
t.text('payload').notNullable();
t.integer('sent_at').notNullable();
t.integer('retries').notNullable().defaultTo(0);
});
console.log("✔ ws_pending_events creata");
}
}
async function ensurePendingEvents() {
const exists = await db.schema.hasTable('pending_events');
if (!exists) {
await db.schema.createTable('pending_events', (t) => {
t.increments('id').primary();
t.string('user_id').notNullable();
t.string('session_id').notNullable();
t.string('device_id').notNullable();
t.string('event_id').notNullable();
t.text('payload').notNullable();
t.integer('created_at').notNullable();
t.unique(['user_id', 'session_id', 'device_id', 'event_id']);
});
console.log("✔ pending_events creata");
}
}
/* ============================================================
🏃 ATTIVITÀ — nuova tabella
============================================================ */
async function ensureActivities() {
const exists = await db.schema.hasTable('activities');
if (!exists) {
await db.schema.createTable('activities', (t) => {
t.string('id').primary();
t.string('user');
t.string('cartella');
t.string('file_name');
t.string('file_extension');
t.string('format');
t.string('display_name');
t.string('route_name');
t.string('title');
t.string('description');
t.string('notes');
t.string('sport');
t.string('activity_type');
t.string('device_manufacturer');
t.string('device_model');
t.string('source_app');
t.bigInteger('start_time_millis');
t.bigInteger('end_time_millis');
t.bigInteger('duration_millis');
t.float('distance_meters');
t.float('elevation_gain_meters');
t.float('elevation_loss_meters');
t.float('min_elevation_meters');
t.float('max_elevation_meters');
t.float('file_distance_meters');
t.float('file_elevation_gain_meters');
t.float('computed_distance_meters');
t.float('computed_elevation_gain_meters');
t.float('average_speed_mps');
t.float('max_speed_mps');
t.float('average_pace_sec_per_km');
t.integer('average_heart_rate');
t.integer('max_heart_rate');
t.integer('average_cadence');
t.integer('max_cadence');
t.integer('calories');
t.integer('power_avg');
t.integer('power_max');
t.integer('point_count');
t.integer('segment_count');
t.integer('lap_count');
t.float('min_latitude');
t.float('max_latitude');
t.float('min_longitude');
t.float('max_longitude');
t.float('center_latitude');
t.float('center_longitude');
t.bigInteger('size_bytes');
t.bigInteger('last_modified_millis');
t.bigInteger('date_added_millis');
t.bigInteger('date_modified_millis');
t.bigInteger('last_scan_millis');
t.boolean('is_favorite');
t.boolean('is_hidden');
t.integer('rating');
t.text('tags');
t.text('track_preview');
t.string('scan_status');
t.text('scan_error');
t.string('path');
t.text('extPath').notNullable().defaultTo('');
t.integer('hybrid').notNullable().defaultTo(0);
t.text('diagnostics');
t.text('location');
t.string('fast_hash');
t.datetime('deleted_at').nullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
});
console.log("✔ activities creata");
}
}
/* ============================================================
ATTIVITÀ — deleted_hard
============================================================ */
async function ensureDeletedHardActivities() {
const exists = await db.schema.hasTable('deleted_hard_activities');
if (!exists) {
await db.schema.createTable('deleted_hard_activities', (t) => {
t.string('id').primary();
t.string('user');
t.datetime('deleted_at').defaultTo(db.fn.now());
});
console.log("✔ deleted_hard_activities creata");
}
}
/* ============================================================
ATTIVITÀ — sync_done
============================================================ */
async function ensureSyncDoneActivities() {
const exists = await db.schema.hasTable('sync_done_activities');
if (!exists) {
await db.schema.createTable('sync_done_activities', (t) => {
t.string('session_id');
t.string('device_id');
t.bigInteger('last_sync');
});
console.log("✔ sync_done_activities creata");
}
}
/* ============================================================
VOLTI
============================================================ */
async function ensureFaces() {
const exists = await db.schema.hasTable('faces');
if (!exists) {
await db.schema.createTable('faces', (t) => {
// ID univoco del volto
t.string('id').primary();
// Foto da cui è stato estratto
t.string('photo_id').notNullable();
t.integer('face_index').notNullable();
// Bounding box nel frame originale
t.float('x').notNullable();
t.float('y').notNullable();
t.float('width').notNullable();
t.float('height').notNullable();
// Confidence del detector SCRFD
t.float('score').notNullable();
// 5 landmark × 2 coordinate
t.float('landmark_0_x');
t.float('landmark_0_y');
t.float('landmark_1_x');
t.float('landmark_1_y');
t.float('landmark_2_x');
t.float('landmark_2_y');
t.float('landmark_3_x');
t.float('landmark_3_y');
t.float('landmark_4_x');
t.float('landmark_4_y');
// 128 × float32 = 512 byte
t.binary('embedding').notNullable();
// Risultato HDBSCAN.
// NULL = non ancora clusterizzato.
t.string('cluster_id').nullable();
// Persona assegnata dall'utente.
// NULL = identità non ancora assegnata.
t.string('person_id').nullable();
// Stato applicativo del volto.
// detected / clustered / assigned
t.string('status').notNullable().defaultTo('detected');
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
// Indici principali
t.index(['photo_id']);
t.index(['cluster_id']);
t.index(['person_id']);
t.index(['status']);
t.unique(['photo_id', 'face_index']);
// Evita due record per lo stesso volto nella stessa foto
// solo se il nostro scanner assegnerà un face_index.
});
console.log("✔ faces creata");
}
}
/* ============================================================
CODA SCANSIONE VOLTI
============================================================ */
async function ensureFaceScanJobs() {
const exists =
await db.schema.hasTable('face_scan_jobs');
if (!exists) {
await db.schema.createTable(
'face_scan_jobs',
(t) => {
t.string('photo_id').primary();
// pending / processing / done / error
t.string('status')
.notNullable()
.defaultTo('pending');
t.integer('attempts')
.notNullable()
.defaultTo(0);
t.integer('rescan_requested')
.notNullable()
.defaultTo(0);
t.integer('faces_detected').nullable();
t.integer('execution_ms').nullable();
t.integer('applied_rotation').nullable();
t.text('last_error').nullable();
t.datetime('started_at').nullable();
t.datetime('completed_at').nullable();
t.datetime('created_at')
.defaultTo(db.fn.now());
t.datetime('updated_at')
.defaultTo(db.fn.now());
t.index(['status']);
t.index(['updated_at']);
},
);
console.log("✔ face_scan_jobs creata");
} else {
const hasRescanRequested =
await db.schema.hasColumn(
'face_scan_jobs',
'rescan_requested',
);
if (!hasRescanRequested) {
await db.schema.alterTable(
'face_scan_jobs',
(t) => {
t.integer('rescan_requested')
.notNullable()
.defaultTo(0);
},
);
console.log(
"✔ aggiunta face_scan_jobs.rescan_requested",
);
}
}
}
/* ============================================================
CLUSTER HDBSCAN
============================================================ */
async function ensureFaceClusters() {
const exists = await db.schema.hasTable('face_clusters');
if (!exists) {
await db.schema.createTable('face_clusters', (t) => {
// ID persistente del cluster
t.string('id').primary();
// Identifica una specifica esecuzione di HDBSCAN.
// Esempio: "initial-2026-09-23T04:20:00"
t.string('run_id').notNullable();
// Label numerica prodotta da HDBSCAN.
// Può essere -1 per noise.
t.integer('label').notNullable();
// Numero di facce nel cluster
t.integer('size').notNullable().defaultTo(0);
// Parametri principali usati per generare il cluster
t.integer('min_samples');
t.integer('min_cluster_size');
// NULL se il cluster non è ancora stato
// associato ad una persona.
t.string('person_id').nullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.index(['run_id']);
t.index(['person_id']);
// Un label può comparire una sola volta
// all'interno dello stesso run.
t.unique(['run_id', 'label']);
});
console.log("✔ face_clusters creata");
}
}
/* ============================================================
PERSONE
============================================================ */
async function ensurePersons() {
const exists = await db.schema.hasTable('persons');
if (!exists) {
await db.schema.createTable('persons', (t) => {
// ID stabile dell'identità applicativa
t.string('id').primary();
// Utente proprietario della libreria
t.string('user').nullable();
// Nome assegnato dall'utente
t.string('name').notNullable();
t.datetime('created_at').defaultTo(db.fn.now());
t.datetime('updated_at').defaultTo(db.fn.now());
t.index(['user']);
t.index(['name']);
});
console.log("✔ persons creata");
}
}
/* ============================================================
INIT
============================================================ */
/* async function init() {
try {
console.log("🔧 init DB...");
await ensurePhotos();
await ensureDeletedHard();
await ensureActivities();
await ensureDeletedHardActivities();
await ensureSyncDoneActivities();
await ensureWsSessions();
await ensureWsPending();
await ensurePendingEvents();
console.log("✅ init completato");
process.exit(0);
} catch (err) {
console.error("❌ errore init:", err);
process.exit(1);
}
}
*/
async function init() {
try {
console.log("🔧 init DB...");
await ensurePhotos();
await ensureDeletedHard();
await ensureActivities();
await ensureDeletedHardActivities();
await ensureSyncDoneActivities();
await ensureWsSessions();
await ensureWsPending();
await ensurePendingEvents();
// Face recognition
await ensureFaces();
await ensureFaceScanJobs();
await ensureFaceScanAutomation(db);
await ensureFaceClusters();
await ensurePersons();
console.log("✅ init completato");
process.exit(0);
} catch (err) {
console.error("❌ errore init:", err);
process.exit(1);
}
}
init();

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

1296
db/init.js.ok Normal file

File diff suppressed because it is too large Load diff

12
db/knex.js Normal file
View file

@ -0,0 +1,12 @@
// db/knex.js
const knex = require('knex');
const db = knex({
client: 'better-sqlite3',
connection: {
filename: './api_v1/database.sqlite'
},
useNullAsDefault: true
});
console.log("📌 DB PATH:", require('path').resolve('./api_v1/database.sqlite'));
module.exports = db;

73
db/photos.js Normal file
View file

@ -0,0 +1,73 @@
// db/photos.js
const knex = require('./knex');
async function insertOrUpdatePhoto(row) {
// row deve contenere almeno id, user, cartella, name, path, thumb1, thumb2, fast_hash, mtimeMs, etc.
const exists = await knex('photos').where({ id: row.id }).first();
if (!exists) {
await knex('photos').insert({
id: row.id,
user: row.user,
cartella: row.cartella,
name: row.name,
path: row.path,
thumb1: row.thumb1 || null,
thumb2: row.thumb2 || null,
mime_type: row.mime_type || null,
width: row.width || null,
height: row.height || null,
rotation: row.rotation || null,
size_bytes: row.size_bytes || null,
mtimeMs: row.mtimeMs || null,
taken_at: row.taken_at || null,
fast_hash: row.fast_hash || null,
created_at: knex.fn.now(),
updated_at: knex.fn.now()
});
} else {
await knex('photos').where({ id: row.id }).update({
path: row.path,
thumb1: row.thumb1 || exists.thumb1,
thumb2: row.thumb2 || exists.thumb2,
mime_type: row.mime_type || exists.mime_type,
width: row.width || exists.width,
height: row.height || exists.height,
rotation: row.rotation || exists.rotation,
size_bytes: row.size_bytes || exists.size_bytes,
mtimeMs: row.mtimeMs || exists.mtimeMs,
taken_at: row.taken_at || exists.taken_at,
fast_hash: row.fast_hash || exists.fast_hash,
updated_at: knex.fn.now(),
deleted_at: null
});
}
}
async function markPhotoDeletedHard(id, user) {
await knex('photos').where({ id, user }).del();
await knex('deleted_hard').insert({ id, user, deleted_at: knex.fn.now() }).catch(() => {});
}
async function getPhotosSince(user, sinceIso, limit = 1000) {
// ritorna foto aggiornate dopo sinceIso (ISO string)
return knex('photos')
.where({ user })
.andWhere('updated_at', '>', sinceIso)
.orderBy('updated_at', 'asc')
.limit(limit)
.select('*');
}
async function getDeletedHardSince(user, sinceIso) {
return knex('deleted_hard')
.where({ user })
.andWhere('deleted_at', '>', sinceIso)
.select('id', 'deleted_at');
}
module.exports = {
insertOrUpdatePhoto,
markPhotoDeletedHard,
getPhotosSince,
getDeletedHardSince
};

View file

@ -0,0 +1,73 @@
// db/photos.js
const knex = require('./knex');
async function insertOrUpdatePhoto(row) {
// row deve contenere almeno id, user, cartella, name, path, thumb1, thumb2, fast_hash, mtimeMs, etc.
const exists = await knex('photos').where({ id: row.id }).first();
if (!exists) {
await knex('photos').insert({
id: row.id,
user: row.user,
cartella: row.cartella,
name: row.name,
path: row.path,
thumb1: row.thumb1 || null,
thumb2: row.thumb2 || null,
mime_type: row.mime_type || null,
width: row.width || null,
height: row.height || null,
rotation: row.rotation || null,
size_bytes: row.size_bytes || null,
mtimeMs: row.mtimeMs || null,
taken_at: row.taken_at || null,
fast_hash: row.fast_hash || null,
created_at: knex.fn.now(),
updated_at: knex.fn.now()
});
} else {
await knex('photos').where({ id: row.id }).update({
path: row.path,
thumb1: row.thumb1 || exists.thumb1,
thumb2: row.thumb2 || exists.thumb2,
mime_type: row.mime_type || exists.mime_type,
width: row.width || exists.width,
height: row.height || exists.height,
rotation: row.rotation || exists.rotation,
size_bytes: row.size_bytes || exists.size_bytes,
mtimeMs: row.mtimeMs || exists.mtimeMs,
taken_at: row.taken_at || exists.taken_at,
fast_hash: row.fast_hash || exists.fast_hash,
updated_at: knex.fn.now(),
deleted_at: null
});
}
}
async function markPhotoDeletedHard(id, user) {
await knex('photos').where({ id, user }).del();
await knex('deleted_hard').insert({ id, user, deleted_at: knex.fn.now() }).catch(() => {});
}
async function getPhotosSince(user, sinceIso, limit = 1000) {
// ritorna foto aggiornate dopo sinceIso (ISO string)
return knex('photos')
.where({ user })
.andWhere('updated_at', '>', sinceIso)
.orderBy('updated_at', 'asc')
.limit(limit)
.select('*');
}
async function getDeletedHardSince(user, sinceIso) {
return knex('deleted_hard')
.where({ user })
.andWhere('deleted_at', '>', sinceIso)
.select('id', 'deleted_at');
}
module.exports = {
insertOrUpdatePhoto,
markPhotoDeletedHard,
getPhotosSince,
getDeletedHardSince
};

703
face_review_endpoints.md Normal file
View file

@ -0,0 +1,703 @@
# API Face Review: riepilogo completo degli endpoint
## Scopo generale
Le API Face Review gestiscono il flusso completo di revisione dei risultati del riconoscimento facciale:
- lettura delle run HDBSCAN;
- visualizzazione dei cluster automatici;
- visualizzazione dei volti classificati come noise;
- generazione protetta delle miniature dei volti;
- creazione e rinomina delle persone;
- assegnazione di uno o più cluster alla stessa persona;
- esclusione di singoli volti da una persona;
- assegnazione singola o multipla dei volti non raggruppati;
- conservazione separata del risultato automatico e della decisione manuale.
Principio fondamentale:
```text
cluster automatico != persona confermata
```
Più cluster possono essere collegati allo stesso `person_id` senza fondere o cancellare i cluster originali.
## Prefisso e autenticazione
Tutti gli endpoint descritti usano il prefisso:
```text
/face-review
```
Le route sono protette dal middleware JWT globale. Il client deve inviare:
```http
Authorization: Bearer <token>
```
Il middleware JWT verifica il token e salva il payload in:
```javascript
req.user
```
Il nome dell'utente autenticato è disponibile in:
```javascript
req.user.name
```
### Risoluzione dell'utente visualizzato
La funzione `resolveFaceReviewUser(req)` applica queste regole:
- un utente normale senza parametro `user` vede esclusivamente i propri dati;
- Admin può specificare `?user=Fabio` per visualizzare la libreria di un altro utente;
- un utente normale non può selezionare un altro utente;
- cluster, volti, fotografie e persone vengono sempre verificati lato backend.
Esempi:
```text
JWT name = Fabio
GET /face-review/context
=> target_user = Fabio
JWT name = Admin
GET /face-review/context?user=Fabio
=> target_user = Fabio
JWT name = Fabio
GET /face-review/context?user=Jessica
=> HTTP 403
```
## Endpoint di sola lettura
### GET /face-review/context
Restituisce il contesto di autenticazione e la run completata più recente dell'utente visualizzato.
Richiesta utente normale:
```http
GET /face-review/context
Authorization: Bearer <token>
```
Richiesta Admin:
```http
GET /face-review/context?user=Fabio
Authorization: Bearer <admin-token>
```
Risposta:
```json
{
"authenticated_user": "Fabio",
"target_user": "Fabio",
"is_admin": false,
"can_select_user": false,
"latest_run_id": "c44546a7-55a8-423c-b3b3-7dd5da671a1d"
}
```
Possibili errori:
- `401` se il token non è valido o manca l'identità;
- `403` se un utente normale prova a selezionare un altro utente;
- `404` se l'utente richiesto non possiede dati riconoscibili.
### GET /face-review/runs
Restituisce la cronologia delle run di clustering per l'utente visualizzato.
```http
GET /face-review/runs
GET /face-review/runs?user=Fabio
```
Ogni run contiene, tra gli altri:
- `id`;
- `user`;
- `status`;
- `scope`;
- `embedding_count`;
- `min_cluster_size`;
- `min_samples`;
- `metric`;
- `selection_method`;
- `engine` e `engine_version`;
- `cluster_count`;
- `clustered_faces`;
- `noise_faces`;
- `noise_percent`;
- date di avvio e completamento.
Risposta:
```json
{
"user": "Fabio",
"runs": [
{
"id": "...",
"status": "done",
"embedding_count": 30,
"cluster_count": 5,
"clustered_faces": 14,
"noise_faces": 16
}
]
}
```
### GET /face-review/clusters
Restituisce i cluster e i relativi volti. Senza `run_id` usa la run completata più recente.
```http
GET /face-review/clusters
GET /face-review/clusters?run_id=<run-id>
GET /face-review/clusters?user=Fabio
```
Per ogni cluster restituisce:
- identificativo e label HDBSCAN;
- dimensione;
- parametri della run;
- eventuale `person_id` e nome della persona;
- elenco dei volti appartenenti al cluster.
Ogni volto include:
- `id`;
- `photo_id` e `photo_name`;
- `face_index`;
- detector score;
- probabilità HDBSCAN;
- stato di revisione;
- rotazione;
- URL protetto della miniatura.
Risposta sintetica:
```json
{
"user": "Fabio",
"run": {
"id": "...",
"status": "done"
},
"clusters": [
{
"id": "...",
"label": 2,
"size": 4,
"person_id": "...",
"person_name": "Jessica",
"faces": []
}
]
}
```
### GET /face-review/clusters/:clusterId
Restituisce un singolo cluster, purché appartenga all'utente autorizzato.
```http
GET /face-review/clusters/<cluster-id>
```
Restituisce i dati del cluster e tutti i suoi volti. Risponde `404` se il cluster non esiste o appartiene a un altro utente.
### GET /face-review/unclustered
Restituisce i volti noise della run più recente che non sono ancora assegnati manualmente a una persona.
La query applica concettualmente:
```sql
m.noise_flag = 1
AND f.person_id IS NULL
```
Richiesta:
```http
GET /face-review/unclustered
```
Quando un noise viene assegnato a una persona:
- `cluster_id` resta `NULL`;
- `label` resta `-1`;
- `noise_flag` resta `1`;
- `person_id` viene valorizzato;
- `review_status` diventa `accepted`;
- il volto non compare più in questo endpoint.
### GET /face-review/persons
Restituisce le persone dell'utente visualizzato.
```http
GET /face-review/persons
```
Risposta:
```json
{
"user": "Fabio",
"persons": [
{
"id": "...",
"user": "Fabio",
"name": "Jessica",
"created_at": "...",
"updated_at": "..."
}
]
}
```
### GET /face-review/persons/:personId
Restituisce il dettaglio di una persona e tutti i volti assegnati, inclusi quelli provenienti da cluster e quelli assegnati manualmente dal noise.
```http
GET /face-review/persons/<person-id>
```
La risposta include:
- dati della persona;
- numero totale di volti;
- numero di cluster collegati;
- elenco dei volti;
- cluster di provenienza, se presente;
- miniatura protetta.
Un volto noise assegnato manualmente viene restituito con:
```text
cluster_id = null
cluster_label = null
```
### GET /face-review/faces/:faceId/thumbnail
Genera una miniatura WebP protetta del volto.
```http
GET /face-review/faces/<encoded-face-id>/thumbnail
GET /face-review/faces/<encoded-face-id>/thumbnail?size=256
```
Dimensione predefinita:
```text
192 x 192 pixel
```
Intervallo accettato per `size`:
```text
64 <= size <= 512
```
Pipeline dell'immagine:
1. recupera il volto e la fotografia filtrandoli per utente;
2. risolve il percorso tramite `resolveFilePathForEntryId()`;
3. applica esclusivamente `photos.rotation`;
4. non applica automaticamente l'orientamento EXIF;
5. aggiunge un margine al bounding box;
6. crea un crop quadrato;
7. ridimensiona in WebP.
Header restituiti:
```http
Content-Type: image/webp
Cache-Control: private, max-age=3600
X-Content-Type-Options: nosniff
```
Il frontend deve caricare la miniatura tramite `fetch()` con Bearer token e trasformarla in Object URL, perché un normale elemento immagine non aggiunge automaticamente l'header Authorization.
## Endpoint di scrittura per le persone
### POST /face-review/persons
Crea una nuova persona per l'utente visualizzato.
```http
POST /face-review/persons
Content-Type: application/json
Authorization: Bearer <token>
```
Body:
```json
{
"name": "Jessica"
}
```
Comportamento:
- normalizza spazi e nome;
- rifiuta nomi vuoti;
- limita il nome a 120 caratteri;
- cerca una persona omonima senza distinguere maiuscole e minuscole;
- se esiste già, restituisce la persona esistente;
- altrimenti crea un UUID e inserisce il record.
Codici:
- `201` se la persona è stata creata;
- `200` se il nome esisteva già;
- `400` se il nome è assente o non valido.
### PATCH /face-review/persons/:personId
Rinomina una persona.
```http
PATCH /face-review/persons/<person-id>
Content-Type: application/json
Authorization: Bearer <token>
```
Body:
```json
{
"name": "Nuovo nome"
}
```
Comportamento:
- verifica che la persona appartenga all'utente;
- normalizza e valida il nome;
- blocca duplicati dello stesso utente;
- aggiorna `updated_at`;
- non modifica volti o cluster.
Possibili errori:
- `400` nome non valido;
- `404` persona non trovata;
- `409` nome già utilizzato da un'altra persona.
### DELETE /face-review/persons/:personId/faces/:faceId
Rimuove un volto da una persona senza cancellare il volto e senza cancellare il cluster automatico.
```http
DELETE /face-review/persons/<person-id>/faces/<encoded-face-id>
Authorization: Bearer <token>
```
Operazioni transazionali:
1. verifica persona, volto e utente;
2. imposta `faces.person_id = NULL`;
3. mantiene `faces.cluster_id` invariato;
4. marca la membership come `excluded`;
5. salva `reviewed_by` e `reviewed_at`;
6. inserisce o aggiorna `person_face_exclusions`.
Un volto rimosso da una persona può essere riassociato successivamente dal cluster o dalla UI appropriata.
## Endpoint di assegnazione cluster-persona
### POST /face-review/clusters/:clusterId/assign-person
Assegna un cluster a una persona esistente e consente di includere o escludere singoli volti.
```http
POST /face-review/clusters/<cluster-id>/assign-person
Content-Type: application/json
Authorization: Bearer <token>
```
Body:
```json
{
"person_id": "<person-id>",
"included_face_ids": [
"face:...:0",
"face:...:0"
]
}
```
Comportamento transazionale:
- verifica che persona e cluster appartengano allo stesso utente;
- verifica che tutti gli ID inclusi appartengano al cluster;
- per i volti inclusi:
- imposta `faces.person_id`;
- marca la membership come `accepted`;
- rimuove eventuali esclusioni precedenti;
- per i volti non inclusi:
- non li assegna alla persona;
- marca la membership come `excluded`;
- registra `person_face_exclusions`;
- imposta `face_clusters.person_id`.
Questo endpoint permette di collegare più cluster alla stessa persona. Per esempio:
```text
Cluster 2 -> Jessica
Cluster 3 -> Jessica
Cluster 4 -> Jessica
```
I cluster restano distinti e conservano la cronologia HDBSCAN.
## Endpoint di assegnazione dei noise
### POST /face-review/faces/:faceId/assign-person
Assegna manualmente un singolo volto a una persona.
```http
POST /face-review/faces/<encoded-face-id>/assign-person
Content-Type: application/json
Authorization: Bearer <token>
```
Body:
```json
{
"person_id": "<person-id>"
}
```
Comportamento:
- verifica persona, volto e utente;
- aggiorna `faces.person_id`;
- non modifica `faces.cluster_id`;
- accetta la membership più recente del volto;
- elimina l'eventuale esclusione precedente dalla stessa persona.
Per un noise:
```text
cluster_id resta NULL
label resta -1
noise_flag resta 1
review_status diventa accepted
```
### POST /face-review/unclustered/assign-person
Assegna più volti noise selezionati alla stessa persona in una singola transazione.
Persona esistente:
```json
{
"person_id": "<person-id>",
"face_ids": [
"face:...:0",
"face:...:1"
]
}
```
Nuova persona:
```json
{
"name": "Nuova persona",
"face_ids": [
"face:...:0",
"face:...:1"
]
}
```
Controlli principali:
- almeno un volto deve essere selezionato;
- deve essere presente `person_id` oppure `name`;
- usa la run completata più recente;
- tutti i volti devono essere noise della run corrente;
- tutti devono appartenere all'utente;
- tutti devono avere `person_id IS NULL`;
- se un solo volto non è valido, l'intera transazione viene annullata.
Per ogni volto valido:
- `faces.person_id` viene impostato;
- `cluster_id` resta invariato, normalmente `NULL`;
- la membership passa ad `accepted`;
- vengono salvati revisore e data;
- eventuali esclusioni pregresse dalla persona vengono eliminate.
Risposta:
```json
{
"ok": true,
"user": "Fabio",
"person": {
"id": "...",
"name": "Jessica"
},
"person_created": false,
"run_id": "...",
"assigned_face_ids": [],
"assigned_count": 3
}
```
## Tabelle coinvolte
### face_cluster_runs
Conserva ogni esecuzione del clustering e i relativi parametri, metriche e stato.
### face_clusters
Conserva i cluster automatici. Il campo `person_id` collega il cluster alla persona confermata senza eliminare il cluster.
### face_cluster_memberships
Conserva l'appartenenza di ogni volto a una run e a un eventuale cluster.
Campi rilevanti:
- `run_id`;
- `cluster_id`;
- `face_id`;
- `label`;
- `probability`;
- `noise_flag`;
- `review_status`;
- `reviewed_by`;
- `reviewed_at`.
Stati di revisione:
```text
suggested
accepted
excluded
```
### faces
Conserva bounding box, embedding e stato corrente del volto.
Distinzione importante:
- `cluster_id` rappresenta il raggruppamento automatico corrente;
- `person_id` rappresenta la persona confermata manualmente.
### persons
Conserva le persone nominate, sempre separate per utente.
### person_face_exclusions
Registra che uno specifico volto non appartiene a una specifica persona. Impedisce di perdere la decisione manuale durante successive revisioni.
## Regole di coerenza
### Volto clusterizzato e accettato
```text
cluster_id = valorizzato
person_id = valorizzato
noise_flag = 0
review_status = accepted
```
### Volto noise non revisionato
```text
cluster_id = NULL
person_id = NULL
label = -1
noise_flag = 1
review_status = suggested
```
### Volto noise assegnato manualmente
```text
cluster_id = NULL
person_id = valorizzato
label = -1
noise_flag = 1
review_status = accepted
```
### Volto escluso da una persona
```text
person_id = NULL
cluster_id = invariato
review_status = excluded
person_face_exclusions = riga presente
```
## Transazioni e sicurezza
Tutte le operazioni di assegnazione o rimozione importanti vengono eseguite in transazione. Se una validazione o una scrittura fallisce, viene eseguito il rollback completo.
Le API non accettano percorsi filesystem dal browser. Le miniature partono dal solo `faceId`, recuperano la fotografia dal database e risolvono internamente il file.
Ogni operazione verifica la coerenza tra:
```text
utente della run
utente della fotografia
utente della persona
utente autenticato o selezionato da Admin
```
## Conversione dei segnaposto in Markdown standard
Questo file usa segnaposto intenzionali:
- `§1` al posto di intestazione livello 1;
- `§2` al posto di intestazione livello 2;
- `§3` al posto di intestazione livello 3;
- ``` al posto della recinzione dei blocchi di codice;
- ` al posto dell'apice rovescio inline.
Esempio di conversione con `sed`, usando caratteri ASCII generati dalla shell:
```bash
hash=$(printf '\043')
tick=$(printf '\140')
sed \
-e "s/^§1 /${hash} /" \
-e "s/^§2 /${hash}${hash} /" \
-e "s/^§3 /${hash}${hash}${hash} /" \
-e "s/```/${tick}${tick}${tick}/g" \
-e "s/`/${tick}/g" \
face_review_endpoints_summary.md \
> face_review_endpoints_standard.md
```
Nota: è importante sostituire prima ``` e poi il singolo `, come mostrato nel comando.

43
faces-tabs.md Normal file
View file

@ -0,0 +1,43 @@
Risultato finale
Con questa modifica avremo:
photos
│
│ photo_id
▼
faces
│
├── embedding BLOB (512 byte)
├── bbox
├── landmarks
├── score
├── cluster_id ────────┐
└── person_id ──────┐ │
│ │
▼ ▼
persons
▲
│
face_clusters
E soprattutto non abbiamo ancora legato artificialmente HDBSCAN all'identità della persona.
Questo ci lascia libero il passaggio successivo:
350k embeddings
↓
GPU kNN TOPK 32
↓
HDBSCAN
↓
face_clusters
↓
utente:
cluster 17 → "Mario"
cluster 42 → "Mario"
↓
persons.id = Mario
che è esattamente la separazione che ci serve per l'applicazione.

154
faces.md Normal file
View file

@ -0,0 +1,154 @@
Perfetto. Ora abbiamo il quadro completo: SQLite tramite better-sqlite3 + Knex.
Per il tuo caso farei una piccola estensione del DB esistente, senza modificare la struttura di photos.
Schema che proporrei
photos
│
└── faces
│
├── embedding BLOB ← 128 × float32
├── cluster_id
└── person_id
│
└── persons
e separatamente:
clusters
Quindi:
photos 1 ─── N faces N ─── 1 clusters
│
└──── N ─── 1 persons
Con una distinzione fondamentale:
cluster_id = risultato dell'algoritmo HDBSCAN
person_id = identità assegnata dall'utente
un person_id può avere più cluster_id
un nuovo volto riconosciuto può avere person_id ma cluster_id = NULL
faces
Per esempio:
faces
-----
id
photo_id
x
y
width
height
embedding
cluster_id
person_id
created_at
updated_at
embedding sarebbe un BLOB SQLite contenente:
128 × float32 = 512 byte
Questo è adatto al tuo motore OpenCL: leggiamo il BLOB e lo trasformiamo direttamente in float*.
clusters
clusters
--------
id
run_id
label
size
created_at
run_id è utile perché HDBSCAN potrebbe essere rieseguito più volte durante la fase iniziale.
persons
persons
-------
id
user
name
created_at
updated_at
Poi aggiungerei una struttura per i prototype embedding, invece di obbligarci ad avere un solo embedding rappresentativo per persona.
Ad esempio:
person 7
├── prototype 1 → volto frontale
├── prototype 2 → profilo
├── prototype 3 → occhiali
└── prototype 4 → barba
Questo diventa molto utile per il riconoscimento incrementale.
---
Una scelta importante: niente embedding dei cluster
Non salverei inizialmente un unico:
cluster.embedding
come rappresentazione obbligatoria.
Abbiamo già tutti i face embedding:
faces.embedding
e possiamo calcolare i rappresentanti quando servono.
Per esempio, dopo che l'utente associa un cluster a una persona, possiamo selezionare alcuni embedding rappresentativi del cluster e inserirli nei prototype della persona.
---
Quindi il flusso completo diventa
┌──────────────┐
│ photos │
└──────┬───────┘
│
face detector
│
▼
┌──────────────┐
│ faces │
│ bbox │
│ embedding │
└──────┬───────┘
│
┌──────────┴──────────┐
│ │
inizializzazione nuove foto
│ │
▼ ▼
HDBSCAN prototype search
│ │
▼ ▼
clusters person_id
│
UI utente
│
▼
persons
E soprattutto questa struttura è compatibile con il benchmark OpenCL che stiamo facendo: il DB rimane il persistente, mentre indices[N×32], distanze, MST ecc. rimangono dati di lavoro GPU/CPU e non devono necessariamente finire in SQLite.
Prima di scrivere init.js
Mi manca solo una verifica: hai già codice che estrae i volti / genera gli embedding RKNN?
Se sì, mandami il file che attualmente gestisce quella parte. Così possiamo progettare faces attorno al formato reale dell'output RKNN e non creare una seconda rappresentazione incompatibile.

245
faces2.md Normal file
View file

@ -0,0 +1,245 @@
Perfetto. Ho guardato direttamente il repository e adesso possiamo progettare l'integrazione sul codice reale, non su ipotesi.
Il repository conferma che la pipeline produce esattamente ciò che ci serve: SCRFD → landmark → alignment 112×112 → SFace → 128 float L2-normalizzati. La Face attuale contiene già score, bbox, 5 landmark e std::vector<float> embedding.
La cosa più importante
Non cambierei il modello RKNN.
Il formato ideale per il DB è già quello che abbiamo:
float embedding[128]
Il codice attuale fa anche già la L2 normalization prima di restituire l'embedding.
Quindi il flusso diventa:
photos.id
│
▼
SCRFD
│
├── bbox
├── confidence
└── 5 landmarks
│
▼
alignment
│
▼
SFace
│
▼
float[128]
│
▼
faces
│
├── HDBSCAN
│
└── recognition
E farei una piccola modifica all'architettura del repository
Attualmente face_recognition.cc è un programma di test: prende una/due immagini e stampa i risultati. Il main() stesso conferma che l'attuale uso è principalmente il confronto fra immagini.
Non lo trasformerei direttamente nel processo DB.
Creerei invece qualcosa del genere:
face-rknn/
├── src/
│ ├── face_recognition.cc ← lasciamo il test attuale
│ ├── face_engine.cc
│ ├── face_engine.h
│ └── face_scan.cc ← nuovo scanner DB
dove FaceEngine espone qualcosa tipo:
struct FaceResult {
float score;
float x1, y1, x2, y2;
Point kps[5];
std::array<float, 128> embedding;
};
class FaceEngine {
public:
bool init(
const char* scrfd_model,
const char* sface_model
);
std::vector<FaceResult> process(
const Image& image
);
};
Così il codice RKNN resta riutilizzabile.
---
DB: ora possiamo definirlo con precisione
Dato che sei su SQLite/better-sqlite3, userei:
faces.embedding → BLOB
con esattamente:
128 * sizeof(float)
= 512 byte
per volto.
E non salverei FP16.
Il modello restituisce FP16, ma il codice attuale li converte già in float e poi normalizza.
Quindi salviamo il risultato finale:
float32[128]
che è esattamente quello che poi il tuo OpenCL può consumare.
Per 350.000 volti:
350000 × 512
≈ 179.2 MB
assolutamente gestibile.
---
Una correzione rispetto a quello che ti avevo detto prima
Avendo visto il codice vero, non introdurrei subito prototypes nel DB.
Prima farei funzionare questa pipeline:
photos
↓
faces
↓
OpenCL kNN
↓
HDBSCAN
↓
clusters
↓
utente assegna persona
↓
persons
Poi, quando implementiamo il riconoscimento incrementale, decidiamo se i prototype debbano essere:
una tabella persistente;
un subset di faces;
oppure un rappresentante/calcolo derivato.
Non voglio aggiungere complessità prima che serva.
---
E c'è un dettaglio molto utile
Il repository usa già:
cosine(a, b)
su embedding L2-normalizzati.
Quindi il tuo benchmark OpenCL che stiamo facendo è perfettamente allineato:
cosine(a,b)
=
dot(a,b)
per vettori normalizzati.
Questa è una buona notizia: non dobbiamo cambiare metrica.
Il tuo OpenCL può quindi ricevere direttamente i BLOB float32[128] e fare il dot product.
---
Quindi io procederei così
Passo 1 — DB
Aggiungiamo a init.js:
faces
face_clusters
persons
senza toccare photos.
Passo 2 — FaceEngine
Separiamo detector + SFace dal main() di test.
Passo 3 — scanner
Prende:
photos
e per ogni foto non ancora processata:
image
↓
faces
↓
embedding
↓
INSERT faces
Passo 4 — GPU
Solo quando abbiamo gli embedding nel DB:
SQLite
↓
float32 array
↓
OpenCL
↓
kNN TOPK=32
Passo 5 — HDBSCAN
Usiamo quel kNN per costruire:
core distance
mutual reachability
MST
condensed tree
cluster
Passo 6 — UI
L'utente vede i cluster e assegna:
cluster 123 → "Mario"
che significa:
faces.cluster_id = 123
faces.person_id = Mario
Poi il sistema incrementale può riconoscere direttamente Mario senza rieseguire HDBSCAN globale.
---
A questo punto non ti chiederei altro codice alla cieca: abbiamo già DB + backend + face engine + formato embedding.
Il prossimo passo concreto è che , compatibile con il tuo better-sqlite3, e subito dopo invece di fare soltanto il test da riga di comando. ti preparo **la modifica di `db/init.js` per le tre tabelle**adattiamo `face-rknn` per produrre i record `faces`

69
faces_cmd_files.md Normal file
View file

@ -0,0 +1,69 @@
s121js/
├── bin/
│ └── face_scan
├── lib/
│ └── librknnrt.so
├── models/
│ ├── SCRFD_500M_KPS_640.rknn
│ └── face_recognition_sface_2021dec.rknn
├── api_v1/
│ └── scanner/
│ └── faceScanner.js
├── db/
├── tools/
│ └── scanPhotoFaces.js
└── public/
cd /home/nvme/dockerdata/prove/s122js
node tools/scanPhotoFaces.js \
e30a1427708cfc8d63a66d369781b15ac6a07b43e13550b1a7625003914503ed
node tools/scanPhotoFaces.js \
2454c04dd9067d2b5288dcca99c5c3eac85660779a97d6165a08ca7c37481070
sqlite3 -header -column \
api_v1/database.sqlite \
"
SELECT
photo_id,
COUNT(*) AS face_count,
COUNT(DISTINCT face_index) AS unique_face_indexes,
MIN(length(embedding)) AS min_embedding_bytes,
MAX(length(embedding)) AS max_embedding_bytes
FROM faces
WHERE photo_id =
'e30a1427708cfc8d63a66d369781b15ac6a07b43e13550b1a7625003914503ed'
GROUP BY photo_id;
"
[✓] FaceEngine riutilizzabile
[✓] face_scan JSON
[✓] eseguibile relocabile
[✓] runtime RKNN relativo
[✓] modelli relativi
[✓] root server calcolata automaticamente
[✓] conversione embedding in BLOB
[✓] inserimento transazionale
[✓] idempotenza
[✓] rotazione 0/90/180/270
[✓] validazione rotazione server/motore
[✓] test positivo con due volti
[✓] gestione corretta di zero volti
photos.rotation
↓
validazione Node.js
↓
face_scan --rotation 0|90|180|270
↓
rotazione RGB
↓
SCRFD + SFace
↓
validazione JSON e rotazione applicata
↓
transazione SQLite
↓
faces.embedding = BLOB float32[128]

22
file_lungo.md Normal file
View file

@ -0,0 +1,22 @@
prettier --write public/js/map/mapboxMarkers.js
wc -l public/js/map/mapboxMarkers.js
sed -n '1,500p' public/js/map/mapboxMarkers.js
sed -n '501,1000p' public/js/map/mapboxMarkers.js
sed -n '1001,1600p' public/js/map/mapboxMarkers.js
pervstampare a video una funzione scritta come
function myfunction { .... } nel file
questa funzione conta le parentesi graffe
awk '
/^[[:space:]]*function[[:space:]]+myFunction[[:space:]]*\{/ {
p=1
n=0
}
p {
print
n += gsub(/\{/, "{")
n -= gsub(/\}/, "}")
if (n == 0) exit
}
' file

Some files were not shown because too many files have changed in this diff Show more