A paper accepted to IEEE Transaction on Pattern Analysis and Machine Intelligence (TPAMI), 2024