twisted endoscopy and the generic packet conjecture

CONJECTURE

TAKUYA KONNO

Abstract. We present a twisted analogue of the result of Rodier and Mœglin-Waldspurger on the dimension of the space of degenerate Whittaker vectors. This allows us to prove that certain twisted endoscopy forGL(n) implies the (local) generic packet conjecture for many classical groups.

1. Introduction

The Whittaker model of an irreducible representation was first introduced by Jacquet- Langlands as a natural local counter part of the Fourier coefficients of automorphic forms on GL(2). Its extension to GL(n) was used by Bernstein-Zelevinsky to study the ir- reducible representations of GL(n) over p-adic fields [4], [21]. Their results played a fundamental role in the study of the Rankin-Selberg L-functions of GL(n) by Jacquet- Piatetskii-Shapiro-Shalika.

Let F be a local field and G a connected reductive quasisplit group over F. Take an F-rational Borel subgroup B of G and write U for its unipotent radical. We can speak of the Whittaker model associated to a character χ of U(F), which is non-degenerate in the sense that its stabilizer in B(F) equals Z_G(F)U(F). Suppose an irreducible smooth representation (π, V) of G(F) isχ-generic, i.e. admits a Whittaker model. Then Shahidi defined a large family of its corresponding automorphicLandε-factors [17]. These factors and their relationships with the harmonic analysis are very important in the modern theory of automorphic representations. Hence one hopes to extend his theory to non- generic representations.

Conjecturally, the set Π_temp(G(F)) of isomorphism classes of irreducible tempered rep- resentations ofG(F) should be partitioned into a disjoint union of finite subsets Π_φ, called L-packets, parametrized by the so-called Langlands parameters φforG. The elements of Π_φ should share the same L and ε-factors which can be directly defined from φ. Thus to extend Shahidi’s definition, it suffices to find a χ-generic element in each Π_φ. This is exactly the assertion of the generic packet conjecture. In the archimedean case this was proved by Vogan [20], so that we assume F to be non-archimedean from now on.

Of course, a definition of temperedL-packets and their fundamental properties are the prerequisite to discuss the conjecture. At present, these assumptions are verified only for GL(n),SL(n) andU(3). The conjecture forGL(n) and SL(n) are due to Bernstein [21].

Gelbart-Rogawski-Soudry proved a beautiful description of the endoscopic L-packets of U(3) in terms of theta liftings, and deduced the conjecture for them [8]. Moreover, Friedberg-Gelbart-Jacquet-Rogawski compared the relative trace formulae of U(3) and

2000Mathematics Subject Classification. Primary 11F70; Secondary 11F85, 22E50.

Partially supported by the Grants-in-Aid for Scientific Research No. 12740018, the Ministry of Edu- cation, Science, Sports and Culture, Japan.

1

GL(3), and completed the generic packet conjecture for U(3) [6], [7]. Their results also include the global counter part.

Their approaches are different from the method of Vogan which relates the genericity to certain growth property of representations. To treat the general case, it is desirable to have a method similar to his. In fact, a result of Rodier [15] relates the genericity of an irreducible representation with the asymptotic behavior of its character around the identity. If a tempered L-packet Π of G is endoscopic, we have a temperedL-packet Π^H of an endoscopic groupH which lifts to Π. In this situation, Shahidi used Rodier’s result to reduce the generic packet conjecture for Π to that for Π^H [17, § 9].

In this note, we present an extended version of Shahidi’s approach. We give a certain twisted version of [15]. Then we apply this to the (still conjectural at present) twisted endoscopic lifting for GL(n) with respect to an outer automorphism θ [2, § 9], which gives liftings of the tempered L-packets of many classical groups to irreducible tempered representations of GL(n). (It seems that Arthur has almost completed this program but under the assumption that the so-called fundamental lemma for any relevant endoscopic data holds.) We deduce the generic packet conjecture for these classical groups from the existence of such twisted endoscopic liftings.

In fact our result is a twisted analogue of the result of Mœglin and Waldspurger [14] on degenerate Whittaker models rather than the non-degenerate case [15]. This facilitates us to consider more interesting examples of twisted characters of non-generic representations (cf. examples contained in§ 2.6).

2. Twisted analogue of a result of Moeglin-Waldspurger

LetGbe a connected reductive group over ap-adic fieldF of odd residual characteristic.

We fix a non-trivial additive character ψ of F, and a non-degenerate G(F)-invariant bilinear form B(, ) on the Lie algebra g(F) ofG(F).

2.1. Degenerate Whittaker models. Recall that a degenerate Whittaker model (or a generalized Gelfand-graev model) is associated to a pair (N, ϕ) [14, I.7] (cf. [11, 2.2]).

Here N ∈ g(F) is a nilpotent element and ϕ : Gm → G is an F-rational homomorphism such that

Ad(ϕ(t))N =t⁻²N, ∀t∈Gm. Let us recall the construction. ϕ gives a gradationg=⊕

i∈Z g_i, where g_i :={X ∈g|Ad(ϕ(t))X =tⁱX, t∈Gm}.

If we write g^N for the centralizer of N in g, then we can take its Ad(ϕ(Gm))-stable complementming: g=m⊕g^N. The above gradation restricts to gradationsm=⊕

i∈Z mi

and g^N =⊕

i∈Z g^N_i . We introduce two unipotent subgroups U := exp(∑

i≥1

g_i)

, V := exp(

g^N₁ +∑

i≥2

g_i) . The character

χ:V(F)∋expX 7−→ψ(B(N, X))∈C¹

is well-defined and stable under Ad(U(F)) [14, I.7 (2)]. B_N(X, Y) := B(N,[X, Y]), (X, Y ∈g) restricts to a non-degenerate alternating form onm. In fact it is a duality between mi and m2−i for i ̸= 1 and an alternating form on m1. Define HN = HN,ϕ to be the Heisenberg group associated to (m1(F), ψ ◦BN) if m1 is not trivial, and C¹ := {z ∈

C|zz¯= 1} otherwise. (ρN,SN) denotes the unique irreducible smooth representation of HN on which the centerC¹ acts by the identity representation (multiplication). We have a homomorphism

p_N :U(F)∋expX 7−→(X₁^m;ψ(B(N, X)))∈ HN,

where X₁^m is the m₁(F)-component of X under the decomposition LieU = m₁ ⊕g^N₁ ⊕

∑

i≥2 g_i. Write ¯ρ_N :=ρ_N ◦p_N.

Now let (π, E) be an admissible representation ofG(F). We have the twisted coinvariant space [3]

EV,χ:=E/E(V, χ), E(V, χ) := Span{π(v)ξ−χ(v)ξ|v ∈V(F), ξ∈E}.

Clearly,U(F) acts by some copy of ¯ρ_N onE_V,χ. Define the space ofdegenerate Whittaker vectors with respect to (N, ϕ) by

WN,ϕ(π) := Hom_U(F)( ¯ρ_N, E_V,χ).

2.2. Action of automorphisms. Let θ be an F-automorphism of finite order ℓ of G.

Then θ is automatically quasi-semisimple and a theorem of Steinberg shows that G^θ is reductive. Write G⁺ for the non-connected group G ⋊⟨θ⟩. We impose that B(, ) is θ-invariant,N ∈g^θ(F) andϕ is G^θ-valued. The complement mofg^N can be chosen (and we do choose) to beθ-stable. Following [1], we write Π(G(F)θ) for the set of isomorphism classes of irreducible admissible representations of G⁺(F) whose restrictions toG(F) are still irreducible. Also write Ξ for the group of characters of π₀(G⁺) =⟨θ⟩.

Since HN is θ-stable (we let θ act trivially on the center C¹), we may form H⁺_N :=

HN ⋊⟨θ⟩. ρ_N is also θ-stable and we fix an isomorphism ρ_N,+(θ) :θ(ρ_N)→^∼ ρ_N satisfying ρ_N,+(θ)^ℓ = id. This extends ρ_N to an irreducible representation ρ_N,+ of H_N⁺.

Take (π, E) ∈ Π(G(F)θ). On the space WN,ϕ(π) of degenerate Whittaker vectors for π|G(F), we define the action of θ by

θ(Φ) :SN

ρN,+(θ)⁻¹

−→ SN

−→Φ E_V,χ−→^π(θ) E_V,χ.

Set U⁺ := U ⋊⟨θ⟩ and extend p_N to U⁺(F) → H⁺_N by p_N(θ) = θ. If we write ρ⁺_N :=

Ind^H

+

HNNρ_N and ¯ρ⁺_N :=ρ⁺_N ◦p_N, then the Mackey theory ρ⁺_N ≃⊕

ζ∈Ξ ζρ_N,+ gives WN,ϕ(π)≃Hom_U(F)(E_V,χ^∗ ,ρ¯^∨_N)≃⊕

ζ∈Ξ

Hom_U⁺_(F)(ζρ¯_N,+, E_V,χ).

Putting WN,ϕ(π)_ζ := Hom_U⁺_(F)(ζρ¯_N,+, E_V,χ), we have θ(Φ) = ζ(θ)Φ for Φ ∈ WN,ϕ(π)_ζ and hence

(2.1) tr(θ|WN,ϕ(π)) =∑

ζ∈Ξ

ζ(θ) dimWN,ϕ(π)_ζ.

2.3. Descent in the twisted case. Let us recall Harish-Chandra’s descent in the twisted case [5, 3.4]. We shall be concerned with the distributions invariant under theθ-conjugation:

Ad_θ(g)x:=gxθ(g)⁻¹, g, x∈G(F).

Writingg(θ) := (1−θ)g⊂g, set

G^θ′(F) :={g ∈G^θ(F)| det(Ad(g)◦θ−1|g(θ, F))̸= 0}, Ωθ := Adθ(G(F))G^θ′(F).

Ωθ is a dense, Adθ(G(F))-invariant subset of G(F). Moreover the map G(F)×G^θ′(F)∋(g, m)7−→Adθ(g)m ∈G(F)

is submersive [16, Prop. 1]. Then [9, Th. 11] asserts the existence of a surjective linear map

C_c^∞(G(F)×G^θ′(F))∋α(g, m)7−→φ(x)∈C_c^∞(Ωθ) such that

∫

G(F)

∫

G^θ(F)

α(g, m)Φ(gmθ(g)⁻¹)dm dg=

∫

G(F)

φ(x)Φ(x)dx

for any Φ∈C_c^∞(G(F)) and fixed invariant measures onG(F) andG^θ(F). We writeD(X) for the space of distributions on aℓ-space X in the sense of [3]. Dual to the above map is

D(Ω_θ)∋T 7−→τ ∈ D(G(F)×G^θ′(F))

given by ⟨τ, α⟩:=⟨T, φ⟩. Of course the mapα7→φ is not injective, but ϕ(m) :=

∫

G(F)

α(g, m)dg∈C_c^∞(G^θ′(F)) is uniquely determined by φ.

Lemma 2.1. For any Adθ(G(F))-invariant T ∈ D(Ωθ), there exists an Ad(G^θ(F))- invariant distribution σ_T ∈ D(G^θ′(F)) such that

⟨T, φ⟩=⟨σT, ϕ⟩, ∀φ∈C_c^∞(Ωθ).

In the ordinary case, this is [9, Lem. 21]. The word-to-word translation to the present case is easy.

2.4. Local expansion twsited characters. Now let (π, E)∈Π(G(F)θ). The distribu- tion

Θπ,θ(f) := tr(π(f)◦π(θ)), f ∈C_c^∞(G(F))

is called thetwisted characterof π. This is clearly Ad_θ(G(F))-invariant and we can apply Lem. 2.1 to have an invariant distribution ϑ_π onG^θ′(F) such that

Θ_π,θ(φ) = ϑ_π(ϕ), ∀φ∈C_c^∞(Ω_θ).

Clozel showed that this ϑ_π is “close to being admissible”. In particular, we have

Theorem 2.2 ([5] Th. 3). Write N(g^θ(F)) for the set of nilpotent Ad(G^θ(F))-orbits in g^θ(F). Then there exist a neighborhood U_π,θ of 0 in g^θ(F) and complex numbers c_o,θ(π), o∈ N(g^θ(F))such that

(2.2) Θ_π,θ(φ) = ∑

o∈N(g^θ(F))

c_o,θ(π)bµ_o(ϕ◦exp)

holds for any φ ∈ C_c^∞(Ω_θ) with supp(ϕ◦exp) ⊂ U_π,θ. Here we have fixed the invariant measures dg and dµ_o(X) on G(F) and o∈ N(g^θ(F)), respectively, following [14, I.8]. µb_o denotes the Fourier transform of the orbital integral on o:

b

µ_o(ϕ◦exp) :=

∫

o

(ϕ\◦exp)(X)dµ_o(X),

where

(ϕ\◦exp)(X) :=

∫

g^θ(F)

ϕ(expY)ψ(B(X, Y))dY.

2.5. A twisted analogue of the result of [14]. Now we can state our main result.

Let (π, E) ∈ Π(G(F)θ). Then we have the character expansion (2.2) and the ordinary character expansion at the identity [10, Th. 5]

(2.3) Θπ(φ) = ∑

O∈N(g(F))

cO(π)bµO(φ◦exp).

As in [14], we write NB(π) for the set of O ∈ N(g(F)) such that c_O(π) ̸= 0. Similarly NB,θ(π) denotes the set of o∈ N(g^θ(F)) such that c_o,θ(π) ̸= 0. We have a partial order O≥O^′ onN(g(F)) defined byO⊃O^′. Write NB(π)^maxfor the set of maximal elements of NB(π) with respect to this order. Also we need the set NB^θ(π)^max of o∈ NB,θ(π) such that the Ad(G(F))-orbitO∈ N(g(F)) containing it belongs toNB(π)^max.

Again following [14], we writeNWh(π) for the set ofO∈ N(g(F)) such thatWN,ϕ(π)̸= 0 for N ∈ O and a suitable choice of ϕ. The subset of maximal elements NWh(π)^max is also defined. Let us writeN_Wh^θ (π)^max for the set of o∈ N(g^θ(F)) such that

(1) The element O∈ N(g(F)) containing o belongs toNWh(π)^max;

(2) For someN ∈o and a suitable ϕ, WN,ϕ is θ-stable and tr(θ|WN,ϕ(π))̸= 0.

By [14, I.16, 17], (1) assures that dimWN,ϕ(π) = cO(π) is finite. Thus the condition (2) makes sense.

Theorem 2.3. Suppose ℓ is prime to the residual characteristic of F, which we assume to be odd.

(1) N_Wh^θ (π)^max =N_B^θ(π)^max ⊂ NB,θ(π)^max.

(2) Let o ∈ NB^θ(π)^max. Then for any choice of N ∈ o, ϕ and m as in § 2.2, we have tr(θ|WN,ϕ(π)) =c_o,θ(π) modulo ℓ-th roots of unity.

Remark 2.4. In (2) above, the ambiguity of ℓ-th roots of unity occurs because our choice of the extension ρ_N,+ of ρ_N is arbitrary. In particular, if (N, ϕ) is chosen so that m₁ = 0, we have the equality without any ambiguity.

2.6. Examples. Let us look at some very basic examples of the above theorem. First we consider the case of G_n := Res_E/FGL(n) for a finite cyclic extension E/F. Take the automorphism θ to be a composite θ₁◦eσ, where θ₁ is any automorphism of GL(n)_E of finite order andeσ is the F-automorphism of Res_E/FGL(n) associated to a generatorσ of Gal(E/F).

Recall the Zelevinsky classification of irreducible admissible representations of G_n(F).

A segment ∆_a,m is the sequence [a+ 1, a+ 2, . . . , a+m] of integers. |∆_a,m| :=m is the length of ∆_a,m. For an irreducible supercuspidal representationρofG_d(F) and a segment

∆_a,m, we write ⟨∆_a,m⟩ρ for the unique irreducible subrepresentation of the parabolically induced representation

ind^G_P^dm(F)

(dm)(F)[(ρ|det|^a+1_E ⊗ρ|det|^a+2_E ⊗ · · · ⊗ρ|det|^a+m_E )⊗111_U(dm)(F)].

Here P_(d^m₎ denotes the standard parabolic subgroup of G_dm associated to the partition (d^m) = (d, d, . . . , d) (m-tuple) and U_(d^m₎ is its unipotent radical. A multi-segment is a finite sequence ∆a,m = [∆_a¹_,m¹, . . . ,∆a^r,m^r] of segments (a = {aⁱ}1≤i≤r, m= {mⁱ}1≤i≤r)

satisfying ai ≥aj for any 1≤i < j ≤r. Then for an irreducible supercuspidal represen- tation ρ of G_d(F), the parabolically induced representation

ind^G_P^d|m|

dm [(⟨∆_a¹_,m¹⟩ρ⊗ · · · ⊗ ⟨∆_a^r_,m^r⟩ρ)⊗111_Udm(F)] admits a unique irreducible subrepresentation ⟨∆_a,m⟩ρ. Here |m| = ∑_r

i=1 mⁱ and dm denotes its partition (dm¹, . . . , dm^r). Now the Zelevinsky classification can be stated as follows.

Proposition 2.5 ([21] Th. 6.1, 7.1). (i) Take a finite family of pairs (∆_aj,mj, ρ_j)_1≤j≤s, where ∆_aj,mj is a multi-segment and ρ_j is an irreducible supercuspidal representation of G_dj(F) (1≤j ≤s) satisfying ρ_i ̸≃ρ_j, i̸=j. Write n= (n₁, . . . , n_s) with n_j :=|m_j| and n:=|n|. Then the parabolically induced module

⟨(∆_aj,mj, ρ_j)_j⟩:= ind^G_P^n(F)

n [(⟨∆_a1,m1⟩ρ1 ⊗ · · · ⊗ ⟨∆_as,ms⟩ρs)⊗111_Un(F)] is irreducible.

(ii) Each irreducible admissible representation π of G_n(F) is of the form ⟨(∆_aj,mj, ρ_j)_j⟩. The family (∆_aj,mj, ρ_j)_1≤j≤s is uniquely determined by π up to permutations.

LetP be a parabolic subgroup of a connected reductive group G and Obe a nilpotent M-orbit in the Lie algebramof a Levi componentM ofP. Then the parabolically induced nilpotent G-orbit i^G_M(O) in g was defined by Lusztig-Spaltenstein [19, II.3]. A nilpotent orbit parabolically induced from the zero orbit i^G_M({0}) is called the Richardson orbit fromP. It is known that the nilpotent orbits ofg_n= LieG_n are all Richardson orbits. In fact, if a nilpotent orbit Ohas the elementary divisors (T −1)ⁿ¹,(T−1)ⁿ², . . . ,(T−1)^ns, then it is the Richardson orbiti^Gtnⁿ({0}) from P^t_n. Here ^tn is the partition corresponding to the transpose of the Young diagram associated to n (the partition dual to n). The following is a recapitulation of [14, Prop. II.2].

Proposition 2.6. (i) NWh(⟨(∆_aj,mj, ρ_j)_j⟩)^max = NB(⟨(∆_aj,mj, ρ_j)_j⟩)^max consists of the Gn(F)-orbit i^Gn

(m^d₁¹,...,m^dss )({0}). Here m^d denotes the partition (m^d₁, . . . , m^d_t) for m = (m₁, . . . , m_t).

(ii) For N ∈i^Gn

(m^d₁¹,...,m^dss )({0}), the space WN,ϕ(⟨(∆_aj,mj, ρ_j)_j⟩) is one dimensional.

Applying Th. 2.3 (2) to this, we have the following.

Corollary 2.7. Suppose an F-automorphism θ of G_n has the finite order prime to the residual characteristic of F. Then for each π ∈Π(G_n(F)θ), the twisted character Θ_π,θ is not zero.

Similarly we deduce the following from the uniqueness of Whittaker models [18].

Corollary 2.8. Suppose an F-automorphism θ of a connected reductive quasisplit F- groupGhas the finite order prime to the residual characteristic of F. Let χbe a character of the unipotent radical U(F) of an F-Borel subgroup B(F) such that Stab(χ, B(F)) = Z_G(F)U(F). Then the twisted character Θ_π,θ of an irreducible χ-generic representation π is not trivial.

Remark 2.9. Both of these follows from the fact that the corresponding space WN,ϕ(π) is one dimensional. Otherwise, even if WN,ϕ(π)̸= 0 the trace of θ on it can be zero. An example of this phenomenon is constructed by Ju-Lee Kim and Piatetskii-Shapiro [12].

3. Twisted endoscopy implies the generic packet conjecture

3.1. Twisted endoscopy problems to be considered. The fundamentals of the gen- eral theory of twisted endoscopy are exploited in [13]. Here we treat some very special but important examples of the theory introduced by Arthur [2, § 9].

Let F be a p-adic field of odd residual characteristic. Recall that a twisted endoscopy problem is attached to a triple (G, θ,a), where G is a connected reductive group over F, θ is a quasi-semisimple automorphism of G and a is a class in H¹(W_F, Z(G)).b a can be considered as the equivalence class of Langlands parameters attached to a character ω of G(F). Then the theory is intended to study those irreducible admissible representations or “packets” Π of G(F) satisfying Π◦θ≃ω⊗Π.

LetE be a quadratic extension ofF orF itself. Writeσ for the generator of the Galois group Γ_E/F of this extension. We shall be concerned with the following triple (L, θ,111).

L:= Res_E/FGL(n). θ:=θn◦σe where θn is the automorphism

θ_n(g) := Ad(I_n)(^tg⁻¹), I_n:=







1

−1 .·^. (−1)ⁿ⁻¹





,

andσedenotes the F-automorphism of Lattached toσ by theF-structure ofGL(n). The sets of endoscopic data which we shall consider are given as follows [2,§ 9].

Case (A): E ̸=F. In this case, we have ^LL=GL(n,C)²⋊ρ_LWF with ρ_L(w)(g₁, g₂) =

{

(g₁, g₂) if w∈W_E, (g₂, g₁) otherwise.

The set of endoscopic data is (G,^LG, s, ξ) whereGis the quasisplit unitary group U_E/F(n) in n variables attached to E/F, s := (111_n,111_n) ∈ Gb and ξ : ^LG ,→ ^LL is given by

ξ(g⋊ρGw) = (g, θ_n(g))⋊ρLw, g ∈Gb =GL(n,C), w ∈W_F.

Case (B): E =F, n = 2m is even. In this case ^LL =GL(n,C)×W_F. The set of data is (G,^LG, s, ξ) whereG=SO(2m+ 1) is the split orthogonal group in 2m+ 1 variables,s := 111_2m and

ξ :^LG∋g×w7−→g×w∈^LL.

HereGb =Sp(m,C) is realized with respect to I_2m.

Case (C): E = F, n = 2m + 1 is odd. The set of data (G,^LG, s, ξ) is given by G=Sp(n), s= 111_2m+1,

ξ :^LG∋g×w7−→g×w∈^LL.

HereGb =SO(2m+ 1,C) is realized with respect to I_2m+1.

Case (D): E = F, n = 2m is even. Let K be a quadratic extension of F or F itself. The set of data is (G,^LG, s, ξ). Here G is the quasisplit orthogonal group which is isomorphic to the split group SO(2m) over K, and on which the non- trivial elementτ of Gal(K/F) (if exists) acts by the unique non-trivial element of

Int(O(2m))/Int(SO(2m)). s:= diag(111m,−111m) and ξ is given by

ξ(g⋊ρGw) =







g×w if w∈WK, g

(

I_m

tIm

)

×w otherwise.

HereGb =SO(2m,C) is realized with respect to (_I ^Im

m ).

3.2. Working hypotheses. We now make some assumptions on some harmonic analytic aspects of the twisted endoscopy for (L, θ,111) and the set of endoscopic data (G,^LG, s, ξ) introduced above.

For any δ ∈ L, we write L^δ,θ for the group of points in L fixed under Ad(δ)◦θ and L_δ,θ for its identity component. δ ∈ L is θ-semisimple if Ad(δ)◦θ induces a semisimple automorphism of the Lie algebra LieLderof the derived group of L. Aθ-semisimpleδ∈L is θ-regular if Lδ,θ is a torus, and strongly θ-regular if L^δ,θ is abelian. We write Lθ,sr(F) for the set of strongly θ-regular elements in L(F). At each δ ∈ L_θ,sr(F) we define the θ-orbital integral by

O_δ,θ(f) :=

∫

L_δ,θ(F)\L(F)

f(g⁻¹δθ(g))dg dt.

Two stronglyθ-regularδ,δ^′ ∈L(F) arestablyθ-conjugate if they areθ-conjugate inL( ¯F).

We define the stable θ-orbital integral at δ∈L_θ,sr(F) by SO_δ,θ(f) := ∑

δ^′ stablyθ-conj. toδ mod. θ-conj.

O_δ′,θ(f).

In [13, Ch. 3], Kottwitz and Shelstad constructed the (strongly regular)norm map, which we denote by NL/G, from the set of stableθ-conjugacy classes inLθ,sr(F) to that of strongly regular stable conjugacy classes in G(F). Also they defined a function ∆_L/G(γ, δ) on G_sr(F)×L_θ,sr(F) called the transfer factor. Of course their construction applies to the most general setting. In our case, we know that

(3.1) ∆_L/G(γ, δ) =

{

1 if γ ∈N_L/G(δ), 0 otherwise.

To define the endoscopic lifting, we need the following conjecture.

Conjecture 3.1 (Transfer conjecture). For f ∈C_c^∞(L(F)), there exists f^G ∈C_c^∞(G(F)) such that

SO_γ(f^G) = ∑

δ

∆_L/G(γ, δ)O_δ,θ(f).

Here δ runs over the θ-conjugacy classes whose norm contains γ.

As opposed to the ordinary (i.e. θ = id) case, we do not have the precise notion of stable distributions in the twisted case. But we assume this in the following. We also have to postulate the existence of discreteL-packets. An irreducible admissible representation π of G(F) is square integrable if it appears discretely in Harish-Chandra’s Plancherel formula for G(F). The set of isomorphism classes of such representations is denoted by Πdisc(G(F)).

Conjecture 3.2. (1) Πdisc(G(F)) is partitioned into a disjoint union of finite sets of representations Π_φ called (discrete) L-packets:

Π_disc(G(F)) = ⨿

φ∈Φdisc(G(F))

Π_φ. (2) There exists a function δ(1,•) : Πφ →C^× such that

Θ_φ := ∑

π∈Πφ

δ(1, π)Θ_π

is a stable distribution.

An irreducible admissible representation ofGis tempered if it contributes non-trivially to the Plancherel formula. Let P = M U be a F-parabolic subgroup of G and τ ∈ Πdisc(M(F)). Then the induced representation ind^G(F_P(F⁾₎[τ⊗111_U(F)] is a direct sum of irre- ducible tempered representations of G(F):

ind^G(F_P(F)⁾[τ ⊗111_U(F)]≃

ℓτ

⊕

i=1

πi(τ).

Moreover, any irreducible tempered representation of G(F) is obtained in this way for some (M, τ) unique up to G(F)-conjugation. Regarding this, we define a tempered L- packet by

Π_φ := ⨿

τ∈Π^M_φ

{π_i(τ)|1≤i≤ℓ_τ},

where Π^M_φ is a discrete L-packet of M. By putting δ(1, π_i(τ)) := δ(1, τ), Conj. 3.2 with Π_disc(G(F)) replaced by the set Π_temp(G(F)) of the isomorphism classes of irreducible tempered representations of G(F) follows.

Finally we say thatπ∈Π(L(F)θ) is θ-discrete if it is tempered and is not induced from aθ-stable tempered representation of a proper Levi subgroup. We write Π_disc(L(F)θ) for the subset ofθ-discrete elements in Π(L(F)θ). Note that eachπ ∈Π_disc(L(F)θ) is generic [21]. Now we can define the twisted endoscopic lifting which we need.

Conjecture 3.3. There should be a bijection ξ from the set Φ_disc(G(F)) of discrete L- packets of G(F) to Π_disc(L(F)θ), which should be characterized by

Θ_ξ(Π),θ(f) = c·Θ_Π(f^G),

for any f ∈ C_c^∞(L(F)) and f^G ∈ C_c^∞(G(F)) as in Conj.3.1. Here c is some non-zero constant.

Although we can be more explicit about the constant c if we adopt the Whittaker normalization of the transfer factor [13, 5.3], but it is not necessary for our purpose.

3.3. TE implies GPC. Now we prove the following.

Theorem 3.4. Suppose Conj. 3.3. Then the generic packet conjecture holds for G.

Writel:= LieL. Forh∈C_c^∞(l(F)) andt ∈F^×, we puth_t(X) :=h(t⁻¹X), (X ∈l(F)).

We assume that the support of f ∈ C_c^∞(L(F)) is sufficiently small so that there exists a neighborhood V of 0 in l(F), on which the exponential map is defined and injective, satisfying suppf ⊂exp(V). Then we can considerf◦exp∈C_c^∞(l(F)). Takingtsufficiently

small, we may define ft∈ C_c^∞(L(F)) by ft◦exp := (f ◦exp)t. Further we might take f and V so that the transferred function f^G satisfies the same condition. We define f_t^G in the same fashion. As in [17, Lem. 9.7], one can deduce from (3.1) the following:

Lemma 3.5. Let f ∈C_c^∞(L(F)) and f^G ∈C_c^∞(G(F)) be as in Conj. 3.1. Suppose that suppf is so small that we can define f_t and f_t^G for sufficiently small t. Then we have

SO_γ(f_t^G2) = ∑

δ

∆_L/G(γ, δ)O_δ,θ(f_t²), for t∈F^× small enough.

Let us prove the theorem. Since ind^G(F_P(F⁾₎[τ⊗111_U(F)] is generic if τ is so, we are reduced to the case of a discrete L-packet Π. Then by Conj. 3.3, we have

Θ_ξ(Π),θ(f) =∑

π∈Π

δ(1, π)Θ_π(f^G).

Suppose that suppf is sufficiently small. Then applying the asymptotic expansions (2.2) and (2.3) to the left and right hand sides respectively, we have

∑

O∈N(l^θ(F))

c_O,θ(ξ(Π))µb_O(f^θ◦exp) = ∑

o∈N(g(F))

∑

π∈Π

c_o(π)bµ_o(f^G◦exp).

Heref^θ ∈C_c^∞(G^θ(F)) is the descent off.

Let o ∈ N(g(F)) and N ∈ o. We say that o is r-regular if the variety B_N of Borel subalgebras of g containingN is r-dimensional. It is a result of Harish-Chandra that

b

µ_o(f_t^G2 ◦exp) =|t|^2r_Fbµ_o(f^G◦exp) for an r-regularo [10]. The same is true forl^θ.

Now recall that ξ(Π) is generic. That is, for any 0-regular nilpotent N and ϕ as in

§ 2.2, we have WN,ϕ(ξ(Π)) ̸= 0. It follows from the uniqueness of the Whittaker model that tr(ξ(Π)(θ)|WN,ϕ(ξ(Π))) = 1, and hencec_O,θ(ξ(Π)) = 1 for the regular nilpotent orbit O. Thus in the equality

∑

O∈N(l^θ(F))

c_O,θ(ξ(Π))µb_O(f_t^θ2 ◦exp) = ∑

o∈N(g(F))

∑

π∈Π

c_o(π)µb_o(f_t^G2 ◦exp),

the terms of order 0 in |t|F on the left hand side is not zero. Thus c_o(π) is not zero for at least one regular o. This combined with [14, I.16, 17] implies the genericity of Π.

References

1. James Arthur, Unipotent automorphic representations: conjectures, Ast´erisque (1989), no. 171-172, 13–71, Orbites unipotentes et repr´esentations, II.

2. , Unipotent automorphic representations: global motivation, Automorphic forms, Shimura varieties, andL-functions, Vol. I (Ann Arbor, MI, 1988), Academic Press, Boston, MA, 1990, pp. 1–

75.

3. I. N. Bernˇste˘ın and A. V. Zelevinski˘ı, Representations of the group GL(n, F), where F is a local non-Archimedean field, Uspehi Mat. Nauk31(1976), no. 3(189), 5–70.

4. I. N. Bernstein and A. V. Zelevinsky,Induced representations of reductivep-adic groups. I, Ann. Sci.

Ecole Norm. Sup. (4)´ 10(1977), no. 4, 441–472.

5. Laurent Clozel, Characters of nonconnected, reductive p-adic groups, Canad. J. Math. 39 (1987), no. 1, 149–167.

6. Solomon Friedberg, Stephen Gelbart, Herv´e Jacquet, and Jonathan Rogawski, Repr´esentations g´en´eriques du groupe unitaire `a trois variables, C. R. Acad. Sci. Paris S´er. I Math. 329 (1999), no. 4, 255–260.

7. Stephen Gelbart, Herve Jacquet, and Jonathan Rogawski, Generic representations for the unitary group in three variables, to appear Israel J. Math.

8. Stephen Gelbart, Jonathan Rogawski, and David Soudry, Endoscopy, theta-liftings, and period inte- grals for the unitary group in three variables, Ann. of Math. (2)145(1997), no. 3, 419–476.

9. Harish-Chandra,Harmonic analysis on reductivep-adic groups, Springer-Verlag, Berlin, 1970, Notes by G. van Dijk, Lecture Notes in Mathematics, Vol. 162.

10. ,Admissible invariant distributions on reductivep-adic groups, Lie theories and their applica- tions (Proc. Ann. Sem. Canad. Math. Congr., Queen’s Univ., Kingston, Ont., 1977), Queen’s Univ., Kingston, Ont., 1978, pp. 281–347. Queen’s Papers in Pure Appl. Math., No. 48.

11. Noriaki Kawanaka, Shintani lifting and Gel^′fand-Graev representations, The Arcata Conference on Representations of Finite Groups (Arcata, Calif., 1986), Amer. Math. Soc., Providence, RI, 1987, pp. 147–163.

12. Ju-Lee Kim and Ilya I. Piatetski-Shapiro,Quadratic base change ofθ10, Israel J. Math.123(2001), 317–340.

13. Robert E. Kottwitz and Diana Shelstad,Foundations of twisted endoscopy, Ast´erisque (1999), no. 255, vi+190.

14. C. Mœglin and J.-L. Waldspurger,Mod`eles de Whittaker d´eg´en´er´es pour des groupesp-adiques, Math.

Z.196(1987), no. 3, 427–452.

15. F. Rodier,Mod`ele de Whittaker et caract`eres de repr´esentations, Non-commutative harmonic analysis (Actes Colloq., Marseille-Luminy, 1974), Springer, Berlin, 1975, pp. 151–171. Lecture Notes in Math., Vol. 466.

16. Fran¸cois Rodier, Int´egrabilit´e locale des caract`eres du groupe GL(n, k) o`u k est un corps local de caract´eristique positive, Duke Math. J. 52(1985), no. 3, 771–792.

17. Freydoon Shahidi, A proof of Langlands’ conjecture on Plancherel measures; complementary series forp-adic groups, Ann. of Math. (2)132(1990), no. 2, 273–330.

18. J.A. Shalika,The multiplicity one theorem forGLn, Ann. of Math. (2)100(1974), 171–193.

19. Nicolas Spaltenstein,Classes unipotentes et sous-groupes de Borel, Springer-Verlag, Berlin, 1982.

20. David A. Vogan, Jr., Gel^′fand-Kirillov dimension for Harish-Chandra modules, Invent. Math. 48 (1978), no. 1, 75–98.

21. A. V. Zelevinsky,Induced representations of reductivep-adic groups. II. On irreducible representations of GL(n), Ann. Sci. ´Ecole Norm. Sup. (4)13(1980), no. 2, 165–210.

Graduate School of Mathematics, Kyushu University, 812-8581 Hakozaki, Higashi-ku, Fukuoka, Japan

E-mail address: takuya@math.kyushu-u.ac.jp

URL:http://knmac.math.kyushu-u.ac.jp/T.Konno.html